Abstract:Extensive studies have shown that many animals’ capability of forming spatial representations for self-localization, path planning, and navigation relies on the functionalities of place and head-direction (HD) cells in the hippocampus. Although there are numerous hippocampal modeling approaches, only a few span the wide functionalities ranging from processing raw sensory signals to planning and action generation. This paper presents a vision-based navigation system that involves generating place and HD cells through learning from visual images, building topological maps based on learned cell representations and performing navigation using hierarchical reinforcement learning. First, place and HD cells are trained from sequences of visual stimuli in an unsupervised learning fashion. A modified Slow Feature Analysis (SFA) algorithm is proposed to learn different cell types in an intentional way by restricting their learning to separate phases of the spatial exploration. Then, to extract the encoded metric information from these unsupervised learning representations, a self-organized learning algorithm is adopted to learn over the emerged cell activities and to generate topological maps that reveal the topology of the environment and information about a robot’s head direction, respectively. This enables the robot to perform self-localization and orientation detection based on the generated maps. Finally, goal-directed navigation is performed using reinforcement learning in continuous state spaces which are represented by the population activities of place cells. In particular, considering that the topological map provides a natural hierarchical representation of the environment, hierarchical reinforcement learning (HRL) is used to exploit this hierarchy to accelerate learning. The HRL works on different spatial scales, where a high-level policy learns to select subgoals and a low-level policy learns over primitive actions to specialize on the selected subgoals. Experimental results demonstrate that our system is able to navigate a robot to the desired position effectively, and the HRL shows a much better learning performance than the standard RL in solving our navigation tasks.

Supporting Information for "Hierarchical planning with deep reinforcement learning for three-dimensional navigation of microrobots in blood vessels"

Hierarchical Planning with Deep Reinforcement Learning for 3D Navigation of Microrobots in Blood Vessels

Hierarchical deep reinforcement learning controlled three-dimensional navigation of microrobots in blood vessels

Mapless Collaborative Navigation for a Multi-Robot System Based on the Deep Reinforcement Learning

Vision-Based Robot Navigation through Combining Unsupervised Learning and Hierarchical Reinforcement Learning

Automatic Navigation for Rat-Robots with Modeling of the Human Guidance

Learning Hierarchical Behavior and Motion Planning for Autonomous Driving.

A Navigation Cognitive System Driven by Hierarchical Spiking Neural Network.

Supporting Information for "Brownian Cargo Capture in Mazes via Intelligent Colloidal Microrobot Swarms"

Micro/Nano Motor Navigation and Localization via Deep Reinforcement Learning

Neuro-Planner: A 3D Visual Navigation Method for MAV With Depth Camera Based on Neuromorphic Reinforcement Learning

Efficient Navigation of Colloidal Robots in an Unknown Environment via Deep Reinforcement Learning

A Method to Plan the Path of a Robot Utilizing Deep Reinforcement Learning and Multi-Sensory Information Fusion

Robot Subgoal-guided Navigation in Dynamic Crowded Environments with Hierarchical Deep Reinforcement Learning

Real-Time Object Navigation With Deep Neural Networks and Hierarchical Reinforcement Learning

Hierarchical Representations and Explicit Memory: Learning Effective Navigation Policies on 3D Scene Graphs using Graph Neural Networks

Deep Reinforcement Learning with Dynamic Graphs for Adaptive Informative Path Planning

Motion Planning and Robust Control for the Endovascular Navigation of a Microrobot

MGRL: Graph neural network based inference in a Markov network with reinforcement learning for visual navigation

Searching from Superior to Inferior: A Hierarchical Relationship for Visual Navigation

Path planning of magnetic microswimmers in high-fidelity simulations of capillaries with deep reinforcement learning