Abstract:Existing large pre-trained models typically map text input to text output in an end-to-end manner, such as ChatGPT, or map a segment of text input to a hierarchy of action decisions, such as OpenVLA. However, humans can simultaneously generate text and actions when receiving specific input signals. For example, a driver can make precise driving decisions while conversing with a friend in the passenger seat. Motivated by this observation, we consider the following question in this work: is it possible to construct a pre-trained model that can provide both language interaction and precise decision-making capabilities in dynamic open scenarios. We provide a definitive answer to this question by developing a new model architecture termed Visual Language Action model for Chatting and Decision Making (VLA4CD), and further demonstrating its performance in challenging autonomous driving tasks. Specifically, we leverage LoRA to fine-tune a pre-trained LLM with data of multiple modalities covering language, visual, and action. Unlike the existing LoRA operations used for LLM fine-tuning, we have designed new computational modules and training cost functions for VLA4CD. These designs enable VLA4CD to provide continuous-valued action decisions while outputting text responses. In contrast, existing LLMs can only output text responses, and current VLA models can only output action decisions. Moreover, these VLA models handle action data by discretizing and then tokenizing the discretized actions, a method unsuitable for complex decision-making tasks involving high-dimensional continuous-valued action vectors, such as autonomous driving. The experimental results on CARLA validate that: (1) our proposed model construction method is effective; (2) compared to the SOTA VLA model, VLA4CD can provide more accurate real-time decision-making while retaining the text interaction capability inherent to LLMs.

LLaRA: Supercharging Robot Learning Data for Vision-Language Policy

OpenVLA: An Open-Source Vision-Language-Action Model

TinyVLA: Towards Fast, Data-Efficient Vision-Language-Action Models for Robotic Manipulation

LLARVA: Vision-Action Instruction Tuning Enhances Robot Learning

Robotic Skill Acquisition via Instruction Augmentation with Vision-Language Models

From Goal-Conditioned to Language-Conditioned Agents via Vision-Language Models

Large Language Models as Generalizable Policies for Embodied Tasks

RL-VLM-F: Reinforcement Learning from Vision Language Foundation Model Feedback

LADEV: A Language-Driven Testing and Evaluation Platform for Vision-Language-Action Models in Robotic Manipulation

CogACT: A Foundational Vision-Language-Action Model for Synergizing Cognition and Action in Robotic Manipulation

Language Reward Modulation for Pretraining Reinforcement Learning

Expanding Frozen Vision-Language Models without Retraining: Towards Improved Robot Perception

RLingua: Improving Reinforcement Learning Sample Efficiency in Robotic Manipulations With Large Language Models

Accelerating Reinforcement Learning of Robotic Manipulations via Feedback from Large Language Models

Large Language Model as a Policy Teacher for Training Reinforcement Learning Agents

How to Build a Pre-trained Multimodal model for Simultaneously Chatting and Decision-making?

LM-Nav: Robotic Navigation with Large Pre-Trained Models of Language, Vision, and Action

VLM-Auto: VLM-based Autonomous Driving Assistant with Human-like Behavior and Understanding for Complex Road Scenes

A3VLM: Actionable Articulation-Aware Vision Language Model

Grounding Language with Visual Affordances over Unstructured Data

Interactive Robot Learning from Verbal Correction