Abstract:The development of decision‐making strategies that can solve complex tasks is of great importance to enable microrobots for emerging biomedical applications. Herein, a hierarchical planning framework with deep reinforcement learning to steer a microrobot to navigate through blood vessels is established. The proposed approach shows robustness and flexibility for accomplishing customized routines in complicated three‐dimensional environments. Designing intelligent microrobots that can autonomously navigate and perform instructed routines in blood vessels, a crowded environment with complexities including Brownian disturbance, concentrated cells, confinement, different flow patterns, and diverse vascular geometries, can offer enormous opportunities and challenges in biomedical applications. Herein, a biological‐agent mimicking a hierarchical control scheme that enables a microrobot to efficiently navigate and execute customizable routines in simplified blood vessel environments is reported. The control scheme consists of two decoupled components: a high‐level controller decomposing complex navigation tasks into short‐ranged, simpler subtasks and a low‐level deep reinforcement learning (DRL) controller responsible for maneuvering microrobots to accomplish subtasks. The proposed DRL controller utilizes 3D convolutional neural networks and is capable of learning control policies directly from raw 3D sensory data. It is shown that such a control scheme achieves effective and robust decision‐making within unseen, diverse complicated environments and offers flexibility for customizable task routines. This study provides a proof of principle for designing intelligent control systems for autonomous navigation in vascular networks for microrobots.

Micro/Nano Motor Navigation and Localization via Deep Reinforcement Learning

Navigating micro- and nano-motors/swimmers with machine learning: Challenges and future directions

Improved SLAM and Motor Imagery Based Navigation Control of a Mobile Robot

Efficient Navigation of Colloidal Robots in an Unknown Environment via Deep Reinforcement Learning

Intelligent Navigation of a Magnetic Microrobot with Model-Free Deep Reinforcement Learning in a Real-World Environment

Robot Localization and Navigation through Predictive Processing using LiDAR

A LiDAR Based End to End Controller for Robot Navigation Using Deep Neural Network

Reinforcement learning of optimal active particle navigation

Hierarchical deep reinforcement learning controlled three-dimensional navigation of microrobots in blood vessels

Hierarchical Planning with Deep Reinforcement Learning for 3D Navigation of Microrobots in Blood Vessels

Learning to Control DC Motor for Micromobility in Real Time with Reinforcement Learning

Smart navigation through a rotating barrier: Deep reinforcement learning with application to size-based separation of active microagents

Active particles using reinforcement learning to navigate in complex motility landscapes

Enabling microrobotic chemotaxis via reset-free hierarchical reinforcement learning

Autonomous Navigation by Mobile Robot with Sensor Fusion Based on Deep Reinforcement Learning

Automatic Navigation for Rat-Robots with Modeling of the Human Guidance

Autonomous 3D positional control of a magnetic microrobot using reinforcement learning

Adaptable navigation of magnetic microrobots

Potential field mechanisms and distributed learning for enhancing the navigation of micro/nanorobot in biomedical environments

Model-Based Reinforcement Learning for Ultrasound-Driven Autonomous Microrobots

Intelligent mobile robot navigation in unknown and complex environment using reinforcement learning technique