Abstract:Enabling robots to learn manipulation tasks by observing human demonstrations remains a major challenge. Recent advances in video captioning tasks provide an end-to-end method to translate demonstration videos into robotic commands. Compared with general video captioning tasks, Video to Command (V2C) task faces two key challenges: (1) How to extract key frames containing fine-grained manipulation actions from demonstration videos that contain a large amount of redundant information; (2) How to significantly improve the accuracy of generated commands so that the V2C method can be applied to real robot tasks. In response to the above problems, we propose a multi-modal framework for robots to learn manipulation tasks from human demonstrations. This framework consists of five components: Text Encoder, Video Encoder, Action Classifier, Keyframe Aligner and Command Decoder. In this framework, we have mainly done two aspects of work: (1) The key frame information of the video is extracted, and the effect of key frame information on improving the translation accuracy of robot commands is analyzed; (2) Based on the video and caption text information, we explore the effect of multimodal information fusion on improving the accuracy of the command generated by the model. Experiments show that our model is significantly superior to the existing methods on the standard metrics of video captioning tasks, such as BLEU_N, METEOR, ROUGE_L, and CIDEr. Among them, the performance of the variant model CGM-V using only video information on BLEU_4 is increased by 0.8%, and that of the variant model CGM-M using multi-modal information on BLEU_4 is significantly increased by 43.7%. Furthermore, our framework, when combined with an affordance detection network and a motion planner, can enable the robot to reproduce the tasks in the demonstration. Our source code and expanded annotations for the IIT-V2C dataset are at https://github.com/yin0816/CGM-M.

VIMA: Robot Manipulation with Multimodal Prompts

Mastering Robot Manipulation with Multimodal Prompts through Pretraining and Multi-task Fine-tuning

VLMimic: Vision Language Models are Visual Imitation Learner for Fine-grained Actions

NaturalVLM: Leveraging Fine-grained Natural Language for Affordance-Guided Visual Manipulation

ManipVQA: Injecting Robotic Affordance and Physically Grounded Information into Multi-Modal Large Language Models

Contrastive Imitation Learning for Language-guided Multi-Task Robotic Manipulation

VLMbench: A Compositional Benchmark for Vision-and-Language Manipulation

Instruction-ViT: Multi-modal prompts for instruction learning in vision transformer

PIVOT: Iterative Visual Prompting Elicits Actionable Knowledge for VLMs

Benchmarking Vision, Language, & Action Models on Robotic Learning Tasks

Multitask Multimodal Prompted Training for Interactive Embodied Task Completion

Instruction-ViT: Multi-Modal Prompts for Instruction Learning in ViT

MO-VLN: A Multi-Task Benchmark for Open-set Zero-Shot Vision-and-Language Navigation

VIRT: Vision Instructed Transformer for Robotic Manipulation

MOKA: Open-World Robotic Manipulation through Mark-Based Visual Prompting

A Multi-modal Framework for Robots to Learn Manipulation Tasks from Human Demonstrations

ProgPrompt: Generating Situated Robot Task Plans using Large Language Models

Vision-Language Foundation Models as Effective Robot Imitators

IMProv: Inpainting-based Multimodal Prompting for Computer Vision Tasks

Manipulate-Anything: Automating Real-World Robots using Vision-Language Models

ProgPrompt: program generation for situated robot task planning using large language models