Abstract:Modular Reinforcement Learning decomposes a monolithic task into several tasks with sub-goals and learns each one in parallel to solve the original problem. Such learning patterns can be traced in the brains of animals. Recent evidence in neuroscience shows that animals utilize separate systems for processing rewards and punishments, illuminating a different perspective for modularizing Reinforcement Learning tasks. MaxPain and its deep variant, Deep MaxPain, showed the advances of such dichotomy-based decomposing architecture over conventional Q-learning in terms of safety and learning efficiency. These two methods differ in policy derivation. MaxPain linearly unified the reward and punishment value functions and generated a joint policy based on unified values; Deep MaxPain tackled scaling problems in high-dimensional cases by linearly forming a joint policy from two sub-policies obtained from their value functions. However, the mixing weights in both methods were determined manually, causing inadequate use of the learned modules. In this work, we discuss the signal scaling of reward and punishment related to discounting factor γ, and propose a weak constraint for signaling design. To further exploit the learning models, we propose a state-value dependent weighting scheme that automatically tunes the mixing weights: hard-max and softmax based on a case analysis of Boltzmann distribution. We focus on maze-solving navigation tasks and investigate how two metrics (pain-avoiding and goal-reaching) influence each other's behaviors during learning. We propose a sensor fusion network structure that utilizes lidar and images captured by a monocular camera instead of lidar-only and image-only sensing. Our results, both in the simulation of three types of mazes with different complexities and a real robot experiment of an L-maze on Turtlebot3 Waffle Pi, showed the improvements of our methods.

Modular Robot Design Synthesis with Deep Reinforcement Learning

Learning Modular Robot Locomotion from Demonstrations

Learning Modular Robot Control Policies

RL and ANN Based Modular Path Planning Controller for Resource-Constrained Robots in the Indoor Complex Dynamic Environment

Hierarchical Gait Generation for Modular Robots Using Deep Reinforcement Learning

Beyond Tabula-Rasa: a Modular Reinforcement Learning Approach for Physically Embedded 3D Sokoban

Optimizing Modular Robot Composition: A Lexicographic Genetic Algorithm Approach

Deep Reinforcement Learning Based Co-Optimization of Morphology and Gait for Small-Scale Legged Robot

On the Exploration of LM-Based Soft Modular Robot Design

Learning Modular Robot Visual-motor Locomotion Policies

A Modular Robotic Arm Configuration Design Method Based on Double DQN with Prioritized Experience Replay

Real-World Dexterous Object Manipulation based Deep Reinforcement Learning

Integrated Modular Solution for Task Oriented Manipulator Configuration Design

CPG-Based Hierarchical Locomotion Control for Modular Quadrupedal Robots Using Deep Reinforcement Learning.

Low-Rank Modular Reinforcement Learning via Muscle Synergy

Modular Architecture for StarCraft II with Deep Reinforcement Learning

Modular deep reinforcement learning from reward and punishment for robot navigation

A Deep Reinforcement Learning Approach to Optimal Morphologies Generation in Reconfigurable Tiling Robots

Decoding modular reconfigurable robots: A survey on mechanisms and design

Task-Driven Computational Framework for Simultaneously Optimizing Design and Mounted Pose of Modular Reconfigurable Manipulators

How Perception, Actuation, and Communication Impact the Emergence of Collective Intelligence in Simulated Modular Robots.