Hierarchical Decision and Control for Continuous Multitarget Problem: Policy Evaluation with Action Delay
Jiangcheng Zhu,Jun Zhu,Zhepei Wang,Shan Guo,Chao Xu
DOI: https://doi.org/10.1109/tnnls.2018.2844466
IF: 14.255
2018-01-01
IEEE Transactions on Neural Networks and Learning Systems
Abstract:This paper proposes a hierarchical decision-making and control algorithm for the shepherd game, the seventh mission in the International Aerial Robotics Competition (IARC). In this game, the agent (a multirotor aerial robot) is required to contact targets (ground vehicles) sequentially and drive them to a certain boundary to earn score. During the game of 10 min, the agent should be fully autonomous without any human interference. Regarding the lower-level controller and dynamics of the agent, each action takes a duration of time to accomplish. Denoted as an action delay, in this paper, this action duration is nonconstant and is related to the final reward. Therefore, the challenging point is making the agent “aware of time” when applying a certain action. We solve this problem by two approaches: deep Q-networks and lookup table. The action delay predictor in the decision-level is fitted by a lower-level controller. Through simulations by the example of the shepherd game, the effectiveness and efficiency of this approach are validated. This paper helps our team winning the first prize in IARC 2017, and keeps the best record of this mission since it was released in 2013.