Abstract:Foundation models (FMs), large deep learning models pre-trained on vast, unlabeled datasets, exhibit powerful capabilities in understanding complex patterns and generating sophisticated outputs. However, they often struggle to adapt to specific tasks. Reinforcement learning (RL), which allows agents to learn through interaction and feedback, offers a compelling solution. Integrating RL with FMs enables these models to achieve desired outcomes and excel at particular tasks. Additionally, RL can be enhanced by leveraging the reasoning and generalization capabilities of FMs. This synergy is revolutionizing various fields, including robotics. FMs, rich in knowledge and generalization, provide robots with valuable information, while RL facilitates learning and adaptation through real-world interactions. This survey paper comprehensively explores this exciting intersection, examining how these paradigms can be integrated to advance robotic intelligence. We analyze the use of foundation models as action planners, the development of robotics-specific foundation models, and the mutual benefits of combining FMs with RL. Furthermore, we present a taxonomy of integration approaches, including large language models, vision-language models, diffusion models, and transformer-based RL models. We also explore how RL can utilize world representations learned from FMs to enhance robotic task execution. Our survey aims to synthesize current research and highlight key challenges in robotic reasoning and control, particularly in the context of integrating FMs and RL--two rapidly evolving technologies. By doing so, we seek to spark future research and emphasize critical areas that require further investigation to enhance robotics. We provide an updated collection of papers based on our taxonomy, accessible on our open-source project website at: <a class="link-external link-https" href="https://github.com/clmoro/Robotics-RL-FMs-Integration" rel="external noopener nofollow">this https URL</a>.

Combining Reinforcement Learning and Causal Models for Robotics Applications

Causal Reinforcement Learning for Optimisation of Robot Dynamics in Unknown Environments

Causal Counterfactuals for Improving the Robustness of Reinforcement Learning

From Semantics to Execution: Integrating Action Planning With Reinforcement Learning for Robotic Causal Problem-Solving

Causal Curiosity: RL Agents Discovering Self-supervised Experiments for Causal Representation Learning

From Continual Learning to Causal Discovery in Robotics

Causal Reinforcement Learning using Observational and Interventional Data

Building Minimal and Reusable Causal State Abstractions for Reinforcement Learning

Multimodal Deep Reinforcement Learning with Auxiliary Task for Obstacle Avoidance of Indoor Mobile Robot

Combining Model-Based $Q$ -Learning with Structural Knowledge Transfer for Robot Skill Learning

Learning Low-Level Causal Relations using a Simulated Robotic Arm

Active Exploration in Bayesian Model-based Reinforcement Learning for Robot Manipulation

Model-Based Reinforcement Learning Inspired by Augmented PD for Robotic Control

Integrating Reinforcement Learning with Foundation Models for Autonomous Robotics: Methods and Perspectives

Nonprehensile Planar Manipulation through Reinforcement Learning with Multimodal Categorical Exploration

Causal Coordinated Concurrent Reinforcement Learning

A Survey on Causal Reinforcement Learning

Causal Meta-Reinforcement Learning for Multimodal Remote Sensing Data Classification

Enhancing Task Performance of Learned Simplified Models via Reinforcement Learning

Joint Optimization of Sensing, Decision-making and Motion-controlling for Autonomous Vehicles: A Deep Reinforcement Learning Approach

Experimental Evaluation of ROS-Causal in Real-World Human-Robot Spatial Interaction Scenarios