Abstract:The development of artificial microscopic robots, like synthetic microswimmers, is one of the state-of-the-art research topics due to their promising biomedical applications. The movement of microswimmers is affected by stringent constraints because of the low Reynolds number of the surrounding environment. Researchers have been working on enhancing microrobots’ propulsion tactics and figuring out new approaches to find optimal propulsion strategies. In this research, we employ a Reinforcement Learning (RL) algorithm, specifically Q-Learning, to train linear-shaped microrobots, comprised of spheres and rods, by introducing an innovative and pioneer coding approach, termed “Basic Coding”, which is utilized to specify states and actions within the RL framework. Basic Coding is a powerful general method that can be employed for different agents in any discrete RL environment. We show how to apply Basic Coding to various microrobots with different geometrical configurations, like a triangular one. Our smart microswimmers, with different numbers of spheres, acquire the knowledge of the optimal propulsion cycle to accomplish large net mechanical displacement without relying on any pre-existing locomotion expertise. The three-sphere linear microrobot recovers the cycle Najafi and Golestanian suggested. The N-sphere microrobots with higher degrees of freedom can find the optimal propulsion cycle within a reasonable number of learning steps and low computational cost utilizing our RL and Basic Coding approach, while the learning step number significantly increases using other methods like Brute-force search. For example, we show this number for the 5-sphere microrobot is 1.19E03 and 8.97E10 steps using our methodology and Brute-force, respectively. Furthermore, our intelligent microrobots can successfully and adaptively find new optimal strategies in indeterministic environments in the presence of uncertainty. Moreover, the effects of learning parameters on our RL agents are investigated in this work.

Smart navigation through a rotating barrier: Deep reinforcement learning with application to size-based separation of active microagents

Micro/Nano Motor Navigation and Localization via Deep Reinforcement Learning

Reinforcement learning of optimal active particle navigation

Active particles using reinforcement learning to navigate in complex motility landscapes

Navigating micro- and nano-motors/swimmers with machine learning: Challenges and future directions

Intelligent Navigation of a Magnetic Microrobot with Model-Free Deep Reinforcement Learning in a Real-World Environment

Efficient Navigation of Colloidal Robots in an Unknown Environment via Deep Reinforcement Learning

Optimal active particle navigation meets machine learning

Smart active particles learn and transcend bacterial foraging strategies

A Reinforcement Learning Approach to Find Optimal Propulsion Strategy for Microrobots Swimming at Low Reynolds Number

Hunting active Brownian particles: Learning optimal behavior

Learning Efficient Navigation in Vortical Flow Fields

Autonomous navigation of smart microswimmers in non-uniform flow fields

Path planning of magnetic microswimmers in high-fidelity simulations of capillaries with deep reinforcement learning

Independent Control and Path Planning of Microswimmers with a Uniform Magnetic Field

Field-Directed Motion, Cargo Capture, and Closed-Loop Controlled Navigation of Microellipsoids

Orientation-dependent propulsion of active Brownian spheres: from self-advection to programmable cluster shapes

Steering undulatory micro-swimmers in a fluid flow through reinforcement learning

Hierarchical Planning with Deep Reinforcement Learning for 3D Navigation of Microrobots in Blood Vessels

Directional self-locomotion of active droplets enabled by nematic environment

Hierarchical deep reinforcement learning controlled three-dimensional navigation of microrobots in blood vessels