MMRo: Are Multimodal LLMs Eligible as the Brain for In-Home Robotics?

Jinming Li,Yichen Zhu,Zhiyuan Xu,Jindong Gu,Minjie Zhu,Xin Liu,Ning Liu,Yaxin Peng,Feifei Feng,Jian Tang

2024-06-28

Abstract:It is fundamentally challenging for robots to serve as useful assistants in human environments because this requires addressing a spectrum of sub-problems across robotics, including perception, language understanding, reasoning, and planning. The recent advancements in Multimodal Large Language Models (MLLMs) have demonstrated their exceptional abilities in solving complex mathematical problems, mastering commonsense and abstract reasoning. This has led to the recent utilization of MLLMs as the brain in robotic systems, enabling these models to conduct high-level planning prior to triggering low-level control actions for task execution. However, it remains uncertain whether existing MLLMs are reliable in serving the brain role of robots. In this study, we introduce the first benchmark for evaluating Multimodal LLM for Robotic (MMRo) benchmark, which tests the capability of MLLMs for robot applications. Specifically, we identify four essential capabilities perception, task planning, visual reasoning, and safety measurement that MLLMs must possess to qualify as the robot's central processing unit. We have developed several scenarios for each capability, resulting in a total of 14 metrics for evaluation. We present experimental results for various MLLMs, including both commercial and open-source models, to assess the performance of existing systems. Our findings indicate that no single model excels in all areas, suggesting that current MLLMs are not yet trustworthy enough to serve as the cognitive core for robots. Our data can be found in <a class="link-external link-https" href="https://mm-robobench.github.io/" rel="external noopener nofollow">this https URL</a>.

Robotics,Computer Vision and Pattern Recognition

What problem does this paper attempt to address?

The paper attempts to address the issue of evaluating the applicability of Multimodal Large Language Models (MLLMs) in home robots and their reliability as the "brains" of robots. Specifically, the research team designed a new benchmark called MMRo to comprehensively assess four key capabilities of MLLMs in robotic applications: perception, task planning, visual reasoning, and safety measurement. Through this benchmark, the researchers aim to reveal whether existing MLLMs are reliable enough to serve as the core processing unit of robots. Experimental results indicate that no single model currently excels in all aspects, suggesting that current MLLMs are not yet sufficient to act as the cognitive core of robots. This finding provides guidance for future improvements and the development of more powerful robotic cognitive systems.

MMRo: Are Multimodal LLMs Eligible as the Brain for In-Home Robotics?

RoboMP$^2$: A Robotic Multimodal Perception-Planning Framework with Multimodal Large Language Models

RoboLLM: Robotic Vision Tasks Grounded on Multimodal Large Language Models

MLLM-Bench: Evaluating Multimodal LLMs with Per-sample Criteria

CLFR-M: Continual Learning Framework for Robots Via Human Feedback and Dynamic Memory

LLM as A Robotic Brain: Unifying Egocentric Memory and Control

Understanding the Role of LLMs in Multimodal Evaluation Benchmarks

MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Models

Understanding Large-Language Model (LLM)-powered Human-Robot Interaction

MMMU-Pro: A More Robust Multi-discipline Multimodal Understanding Benchmark

Chat with the Environment: Interactive Multimodal Perception Using Large Language Models

MHRC: Closed-loop Decentralized Multi-Heterogeneous Robot Collaboration with Large Language Models

Large Language Models for Robotics: Opportunities, Challenges, and Perspectives

The Labyrinth of Links: Navigating the Associative Maze of Multi-modal LLMs

MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGI

What is the Visual Cognition Gap between Humans and Multimodal LLMs?

PCA-Bench: Evaluating Multimodal Large Language Models in Perception-Cognition-Action Chain

MMT-Bench: A Comprehensive Multimodal Benchmark for Evaluating Large Vision-Language Models Towards Multitask AGI

RoboMM: All-in-One Multimodal Large Model for Robotic Manipulation