Abstract:Reinforcement learning (RL) is a promising approach for solving robotic manipulation tasks. However, it is challenging to apply the RL algorithms directly in the real world. For one thing, RL is data-intensive and typically requires millions of interactions with environments, which are impractical in real scenarios. For another, it is necessary to make heavy engineering efforts to design reward functions manually. To address these issues, we leverage foundation models in this paper. We propose Reinforcement Learning with Foundation Priors (RLFP) to utilize guidance and feedback from policy, value, and success-reward foundation models. Within this framework, we introduce the Foundation-guided Actor-Critic (FAC) algorithm, which enables embodied agents to explore more efficiently with automatic reward functions. The benefits of our framework are threefold: (1) \textit{sample efficient}; (2) \textit{minimal and effective reward engineering}; (3) \textit{agnostic to foundation model forms and robust to noisy priors}. Our method achieves remarkable performances in various manipulation tasks on both real robots and in simulation. Across 5 dexterous tasks with real robots, FAC achieves an average success rate of 86\% after one hour of real-time learning. Across 8 tasks in the simulated Meta-world, FAC achieves 100\% success rates in 7/8 tasks under less than 100k frames (about 1-hour training), outperforming baseline methods with manual-designed rewards in 1M frames. We believe the RLFP framework can enable future robots to explore and learn autonomously in the physical world for more tasks. Visualizations and code are available at \url{<a class="link-external link-https" href="https://yewr.github.io/rlfp" rel="external noopener nofollow">this https URL</a>}.

RLNN: A force perception algorithm using reinforcement learning

Variable Force Tracking Control Via Online Learning

A Robotic Auto-Focus System based on Deep Reinforcement Learning

Reinforcement Learning with Foundation Priors: Let the Embodied Agent Efficiently Learn on Its Own

Learning Force Control for Contact-Rich Manipulation Tasks With Rigid Position-Controlled Robots

Data Informed Residual Reinforcement Learning for High-Dimensional Robotic Tracking Control

Foundation Reinforcement Learning: Towards Embodied Generalist Agents with Foundation Prior Assistance

RLAfford: End-to-End Affordance Learning for Robotic Manipulation

A Path-Integral-Based Reinforcement Learning Algorithm for Path Following of an Autoassembly Mobile Robot

A Residual Reinforcement Learning Method for Robotic Assembly Using Visual and Force Information

Fusing Vision and Force: A Framework of Reinforcement Learning for Elastic Peg-in-Hole Assembly

Reinforcement learning for robot research: A comprehensive review and open issues

A Contact-Safe Reinforcement Learning Framework for Contact-Rich Robot Manipulation

Learning to Regrasp Using Visual–Tactile Representation-Based Reinforcement Learning

Joint Optimization of Sensing, Decision-making and Motion-controlling for Autonomous Vehicles: A Deep Reinforcement Learning Approach

Learning to Discover Task-Relevant Features for Interpretable Reinforcement Learning

Focus-Then-Decide: Segmentation-Assisted Reinforcement Learning

Feature selection and feature learning for high-dimensional batch reinforcement learning: A survey

Interaction Force Measurement of Parallel Robots Based on Structure-Integrated Force Sensors Using Interpretable Linear Neural Networks

Learning Controllable Elements Oriented Representations for Reinforcement Learning