Abstract:Enabling robots to learn manipulation tasks by observing human demonstrations remains a major challenge. Recent advances in video captioning tasks provide an end-to-end method to translate demonstration videos into robotic commands. Compared with general video captioning tasks, Video to Command (V2C) task faces two key challenges: (1) How to extract key frames containing fine-grained manipulation actions from demonstration videos that contain a large amount of redundant information; (2) How to significantly improve the accuracy of generated commands so that the V2C method can be applied to real robot tasks. In response to the above problems, we propose a multi-modal framework for robots to learn manipulation tasks from human demonstrations. This framework consists of five components: Text Encoder, Video Encoder, Action Classifier, Keyframe Aligner and Command Decoder. In this framework, we have mainly done two aspects of work: (1) The key frame information of the video is extracted, and the effect of key frame information on improving the translation accuracy of robot commands is analyzed; (2) Based on the video and caption text information, we explore the effect of multimodal information fusion on improving the accuracy of the command generated by the model. Experiments show that our model is significantly superior to the existing methods on the standard metrics of video captioning tasks, such as BLEU_N, METEOR, ROUGE_L, and CIDEr. Among them, the performance of the variant model CGM-V using only video information on BLEU_4 is increased by 0.8%, and that of the variant model CGM-M using multi-modal information on BLEU_4 is significantly increased by 43.7%. Furthermore, our framework, when combined with an affordance detection network and a motion planner, can enable the robot to reproduce the tasks in the demonstration. Our source code and expanded annotations for the IIT-V2C dataset are at https://github.com/yin0816/CGM-M.

Towards Effective Utilization of Mixed-Quality Demonstrations in Robotic Manipulation via Segment-Level Selection and Optimization

Learning Robot Manipulation Skills from Human Demonstration Videos Using Two-Stream 2-D/3-D Residual Networks with Self-Attention

Human Demonstration Trajectory Refinement for Redundant Manipulators.

Learning Generalizable 3D Manipulation With 10 Demonstrations

Control-oriented modeling of flight demonstrations for quadrotors using higher-order statistics and dynamic movement primitives

Leveraging Locality to Boost Sample Efficiency in Robotic Manipulation

Screw Geometry Meets Bandits: Incremental Acquisition of Demonstrations to Generate Manipulation Plans

AdaDemo: Data-Efficient Demonstration Expansion for Generalist Robotic Agent

Constrained-Space Optimization and Reinforcement Learning for Complex Tasks

Manipulate-Anything: Automating Real-World Robots using Vision-Language Models

Giving Robots a Hand: Learning Generalizable Manipulation with Eye-in-Hand Human Video Demonstrations

Skill Learning in Robot-Assisted Micro-Manipulation Through Human Demonstrations with Attention Guidance

Multiple Interactions Made Easy (MIME): Large Scale Demonstrations Data for Imitation

Learning Fine Pinch-Grasp Skills using Tactile Sensing from A Few Real-world Demonstrations

Data Scaling Laws in Imitation Learning for Robotic Manipulation

SWBT: Similarity Weighted Behavior Transformer with the Imperfect Demonstration for Robotic Manipulation

SkillMimicGen: Automated Demonstration Generation for Efficient Skill Learning and Deployment

A Multi-modal Framework for Robots to Learn Manipulation Tasks from Human Demonstrations

Learning Robotic Manipulation from Demonstrations by Combining Deep Generative Model and Dynamic Control System

Reinforcement learning with Demonstrations from Mismatched Task under Sparse Reward

DexMimicGen: Automated Data Generation for Bimanual Dexterous Manipulation via Imitation Learning