Abstract:Today's largest foundation models have increasingly general capabilities, yet when used as agents, they often struggle with simple reasoning and decision-making tasks, even though they possess good factual knowledge of the task and how to solve it. In this paper, we present a benchmark to pressure-test these models' multimodal decision-making capabilities in the very long-context regime (up to one million tokens) and investigate whether they can learn from a large number of expert demonstrations in their context. We evaluate a wide range of state-of-the-art frontier models as policies across a battery of simple interactive decision-making tasks: playing tic-tac-toe, chess, and Atari, navigating grid worlds, solving crosswords, and controlling a simulated cheetah. We measure the performance of Claude 3.5 Sonnet, Gemini 1.5 Flash, Gemini 1.5 Pro, GPT-4o, o1-mini, and o1-preview under increasing amounts of expert demonstrations in the context $\unicode{x2013}$ from no demonstrations up to 512 full episodes, pushing these models' multimodal long-context reasoning capabilities to their limits. Across our tasks, today's frontier models rarely manage to fully reach expert performance, showcasing the difficulty of our benchmark. Presenting more demonstrations often has little effect, but some models steadily improve with more demonstrations on a few tasks. We investigate the effect of encoding observations as text or images and the impact of chain-of-thought prompting. Overall, our results suggest that even today's most capable models often struggle to imitate desired behavior by generalizing purely from in-context demonstrations. To help quantify the impact of other approaches and future innovations aiming to tackle this problem, we open source our benchmark that covers the zero-, few-, and many-shot regimes in a unified evaluation.

Choice-75: A Dataset on Decision Branching in Script Learning

What do Large Language Models Learn about Scripts?

MULTISCRIPT: Multimodal Script Learning for Supporting Open Domain Everyday Tasks

LMAct: A Benchmark for In-Context Imitation Learning with Long Multimodal Demonstrations

MCScript: A Novel Dataset for Assessing Machine Comprehension Using Script Knowledge

Structured, flexible, and robust: benchmarking and improving large language models towards more human-like behavior in out-of-distribution reasoning tasks

Take a Break in the Middle: Investigating Subgoals towards Hierarchical Script Generation

AgentSense: Benchmarking Social Intelligence of Language Agents through Interactive Scenarios

Learning to Reason in Round-based Games: Multi-task Sequence Generation for Purchasing Decision Making in First-person Shooters

Distilling Script Knowledge from Large Language Models for Constrained Language Planning

HAZARD Challenge: Embodied Decision Making in Dynamically Changing Environments

Modeling Event-Pair Relations in External Knowledge Graphs for Script Reasoning.

SimulBench: Evaluating Language Models with Creative Simulation Tasks

GameBench: Evaluating Strategic Reasoning Abilities of LLM Agents

Decision-Oriented Dialogue for Human-AI Collaboration

Scheherazade: Evaluating Chain-of-Thought Math Reasoning in LLMs with Chain-of-Problems

Hierarchical Decision Making by Generating and Following Natural Language Instructions

Case Study: Testing Model Capabilities in Some Reasoning Tasks

Evaluating Shutdown Avoidance of Language Models in Textual Scenarios

Non-Sequential Graph Script Induction via Multimedia Grounding

Mini-BEHAVIOR: A Procedurally Generated Benchmark for Long-horizon Decision-Making in Embodied AI