Int-HRL: Towards Intention-based Hierarchical Reinforcement Learning

Anna Penzkofer,Simon Schaefer,Florian Strohm,Mihai Bâce,Stefan Leutenegger,Andreas Bulling

2023-06-20

Abstract:While deep reinforcement learning (RL) agents outperform humans on an increasing number of tasks, training them requires data equivalent to decades of human gameplay. Recent hierarchical RL methods have increased sample efficiency by incorporating information inherent to the structure of the decision problem but at the cost of having to discover or use human-annotated sub-goals that guide the learning process. We show that intentions of human players, i.e. the precursor of goal-oriented decisions, can be robustly predicted from eye gaze even for the long-horizon sparse rewards task of Montezuma's Revenge - one of the most challenging RL tasks in the Atari2600 game suite. We propose Int-HRL: Hierarchical RL with intention-based sub-goals that are inferred from human eye gaze. Our novel sub-goal extraction pipeline is fully automatic and replaces the need for manual sub-goal annotation by human experts. Our evaluations show that replacing hand-crafted sub-goals with automatically extracted intentions leads to a HRL agent that is significantly more sample efficient than previous methods.

Machine Learning

What problem does this paper attempt to address?

The paper aims to address the following issues:

1. **Improving the sample efficiency of Reinforcement Learning (RL)**: Although deep reinforcement learning outperforms humans in many tasks, training these models requires a large amount of data. To overcome this limitation, the paper proposes a new method to automatically identify sub-goals in Hierarchical Reinforcement Learning (HRL).

2. **Reducing the need for manual annotation**: Traditional HRL methods often require experts to manually annotate meaningful sub-goals, which is not only time-consuming but also difficult to generalize to other tasks or games. Therefore, the paper proposes a method to automatically extract sub-goals from human eye movement behavior.

Specifically, the paper infers the player's intentions by predicting their eye movement trajectories and uses these intentions as sub-goals. Experimental results show that this method significantly improves sample efficiency and reduces the need for expert manual annotation. Particularly in the Montezuma’s Revenge game of the Atari 2600 game suite, the HRL agent using this method can complete the first level with only 625K training steps, more than tripling the efficiency compared to previous methods. Moreover, this method requires no human intervention, can automatically extract sub-goals from eye movement data, and verifies goal achievement through memory state mapping, thus achieving end-to-end automated training.