Hierarchical Control and Learning of a Foraging CyberOctopus

Chia-Hsien Shih,Noel Naughton,Udit Halder,Heng-Sheng Chang,Seung Hyun Kim,Rhanor Gillette,Prashant G. Mehta,Mattia Gazzola

DOI: https://doi.org/10.1002/aisy.202300088

IF: 7.298

2023-06-24

Advanced Intelligent Systems

Abstract:This article proposes a hierarchical framework for coordinating multiple soft arms inspired by octopus neurophysiology. By decomposing control into high‐level decision‐making, low‐level motor activation, and local reflexive behaviors via sensory feedback, it utilizes a mixed‐modes approach with reinforcement learning for decision‐making and energy shaping for motor execution. The approach is successfully deployed in increasingly challenging foraging scenarios, demonstrating its viability. Inspired by the unique neurophysiology of the octopus, a hierarchical framework is proposed that simplifies the coordination of multiple soft arms by decomposing control into high‐level decision‐making, low‐level motor activation, and local reflexive behaviors via sensory feedback. When evaluated in the illustrative problem of a model octopus foraging for food, this hierarchical decomposition results in significant improvements relative to end‐to‐end methods. Performance is achieved through a mixed‐modes approach, whereby qualitatively different tasks are addressed via complementary control schemes. Herein, model‐free reinforcement learning is employed for high‐level decision‐making, while model‐based energy shaping takes care of arm‐level motor execution. To render the pairing computationally tenable, a novel neural network energy shaping (NN‐ES) controller is developed, achieving accurate motions with time‐to‐solutions 200 times faster than previous attempts. The hierarchical framework is then successfully deployed in increasingly challenging foraging scenarios, including an arena littered with obstacles in 3D space, demonstrating the viability of the approach.