Abstract:Large Language models (LLMs) have shown remarkable success in assisting robot learning tasks, i.e., complex household planning. However, the performance of pretrained LLMs heavily relies on domain-specific templated text data, which may be infeasible in real-world robot learning tasks with image-based observations. Moreover, existing LLMs with text inputs lack the capability to evolve with non-expert interactions with environments. In this work, we introduce a novel learning paradigm that generates robots' executable actions in the form of text, derived solely from visual observations, using language-based summarization of these observations as the connecting bridge between both domains. Our proposed paradigm stands apart from previous works, which utilized either language instructions or a combination of language and visual data as inputs. Moreover, our method does not require oracle text summarization of the scene, eliminating the need for human involvement in the learning loop, which makes it more practical for real-world robot learning tasks. Our proposed paradigm consists of two modules: the SUM module, which interprets the environment using visual observations and produces a text summary of the scene, and the APM module, which generates executable action policies based on the natural language descriptions provided by the SUM module. We demonstrate that our proposed method can employ two fine-tuning strategies, including imitation learning and reinforcement learning approaches, to adapt to the target test tasks effectively. We conduct extensive experiments involving various SUM/APM model selections, environments, and tasks across 7 house layouts in the VirtualHome environment. Our experimental results demonstrate that our method surpasses existing baselines, confirming the effectiveness of this novel learning paradigm.

Service skill improvement for home robots: Autonomous generation of action sequence based on reinforcement learning

Learning Robot Manipulation Skills from Human Demonstration Videos Using Two-Stream 2-D/3-D Residual Networks with Self-Attention

Instruction Parsing and Action Sequence Generation System for Home Service Task.

Autonomous Generation of Service Strategy for Household Tasks: A Progressive Learning Method with A Priori Knowledge and Reinforcement Learning

Learning Efficient Representations for Goal-conditioned Reinforcement Learning Via Tabu Search

Learning to Transform Service Instructions into Actions with Reinforcement Learning and Knowledge Base

A Method For Knowledge Construction From Natural Language Based On Reinforcement Learning

Hybrid Reinforcement Learning Based on Human Preference and Advice for Efficient Robot Skill Learning

Autonomous Planning of Service Robot Based on Natural Language Tasks in Intelligent Space

Learning and Development of Home Service Robots' Service Cognition Based on a Learning Mechanism

CLFR-M: Continual Learning Framework for Robots Via Human Feedback and Dynamic Memory

Autonomous Improvement of Instruction Following Skills via Foundation Models

Embodied Executable Policy Learning with Language-based Scene Summarization

Learning from User-driven Events to Generate Automation Sequences.

Home service robot task planning using semantic knowledge and probabilistic inference

Applying Learning-from-observation to household service robots: three common-sense formulation

Style-transfer based Speech and Audio-visual Scene Understanding for Robot Action Sequence Acquisition from Videos

LLM-based Robot Task Planning with Exceptional Handling for General Purpose Service Robots

Robot Skill Acquisition in Assembly Process Using Deep Reinforcement Learning

Developing high-level cognitive functions for service robots