Abstract:As general-purpose tools, Large Language Models (LLMs) must often reason about everyday physical environments. In a question-and-answer capacity, understanding the interactions of physical objects may be necessary to give appropriate responses. Moreover, LLMs are increasingly used as reasoning engines in agentic systems, designing and controlling their action sequences. The vast majority of research has tackled this issue using static benchmarks, comprised of text or image-based questions about the physical world. However, these benchmarks do not capture the complexity and nuance of real-life physical processes. Here we advocate for a second, relatively unexplored, approach: 'embodying' the LLMs by granting them control of an agent within a 3D environment. We present the first embodied and cognitively meaningful evaluation of physical common-sense reasoning in LLMs. Our framework allows direct comparison of LLMs with other embodied agents, such as those based on Deep Reinforcement Learning, and human and non-human animals. We employ the Animal-AI (AAI) environment, a simulated 3D virtual laboratory, to study physical common-sense reasoning in LLMs. For this, we use the AAI Testbed, a suite of experiments that replicate laboratory studies with non-human animals, to study physical reasoning capabilities including distance estimation, tracking out-of-sight objects, and tool use. We demonstrate that state-of-the-art multi-modal models with no finetuning can complete this style of task, allowing meaningful comparison to the entrants of the 2019 Animal-AI Olympics competition and to human children. Our results show that LLMs are currently outperformed by human children on these tasks. We argue that this approach allows the study of physical reasoning using ecologically valid experiments drawn directly from cognitive science, improving the predictability and reliability of LLMs.

Integration of LLMs and the Physical World: Research and Application

Penetrative AI: Making LLMs Comprehend the Physical World

Large Language Models for Robotics: Opportunities, Challenges, and Perspectives

IoT-LLM: Enhancing Real-World IoT Task Reasoning with Large Language Models

Large Language Models for Robotics: A Survey

A Survey on Integration of Large Language Models with Intelligent Robots

When IoT Meet LLMs: Applications and Challenges

Opportunities and Challenges of Applying Large Language Models in Building Energy Efficiency and Decarbonization Studies: An Exploratory Overview

Integration of Large Language Models and Federated Learning

A little less conversation, a little more action, please: Investigating the physical common-sense of LLMs in a 3D embodied environment

Grounding Large Language Models In Embodied Environment With Imperfect World Models

Language Models Meet World Models: Embodied Experiences Enhance Language Models

The Development of LLMs for Embodied Navigation

When LLMs step into the 3D World: A Survey and Meta-Analysis of 3D Tasks via Multi-modal Large Language Models

Applying Large Language Models for Intelligent Industrial Automation

A General-Purpose Device for Interaction with LLMs

LLM4RL: Enhancing Reinforcement Learning with Large Language Models

Understanding User Experience in Large Language Model Interactions

Large Language Models as Urban Residents: An LLM Agent Framework for Personal Mobility Generation