Abstract:The advancements of language language models (LLMs) have piqued growing interest in developing LLM-based language agents to automate scientific discovery end-to-end, which has sparked both excitement and skepticism about the true capabilities of such agents. In this work, we argue that for an agent to fully automate scientific discovery, it must be able to complete all essential tasks in the workflow. Thus, we call for rigorous assessment of agents on individual tasks in a scientific workflow before making bold claims on end-to-end automation. To this end, we present ScienceAgentBench, a new benchmark for evaluating language agents for data-driven scientific discovery. To ensure the scientific authenticity and real-world relevance of our benchmark, we extract 102 tasks from 44 peer-reviewed publications in four disciplines and engage nine subject matter experts to validate them. We unify the target output for every task to a self-contained Python program file and employ an array of evaluation metrics to examine the generated programs, execution results, and costs. Each task goes through multiple rounds of manual validation by annotators and subject matter experts to ensure its annotation quality and scientific plausibility. We also propose two effective strategies to mitigate data contamination concerns. Using our benchmark, we evaluate five open-weight and proprietary LLMs, each with three frameworks: direct prompting, OpenHands, and self-debug. Given three attempts for each task, the best-performing agent can only solve 32.4% of the tasks independently and 34.3% with expert-provided knowledge. These results underscore the limited capacities of current language agents in generating code for data-driven discovery, let alone end-to-end automation for scientific research.

ChinaTravel: A Real-World Benchmark for Language Agents in Chinese Travel Planning

TravelPlanner: A Benchmark for Real-World Planning with Language Agents

TravelAgent: An AI Assistant for Personalized Travel Planning

Smart Language Agents in Real-World Planning

Can We Rely on LLM Agents to Draft Long-Horizon Plans? Let's Take TravelPlanner as an Example

Towards Full Delegation: Designing Ideal Agentic Behaviors for Travel Planning

AgentSense: Benchmarking Social Intelligence of Language Agents through Interactive Scenarios

Revealing the Barriers of Language Agents in Planning

Be More Real: Travel Diary Generation Using LLM Agents and Individual Profiles

Large Language Models Can Solve Real-World Planning Rigorously with Formal Verification Tools

To the Globe (TTG): Towards Language-Driven Guaranteed Travel Planning

NATURAL PLAN: Benchmarking LLMs on Natural Language Planning

Ask-before-Plan: Proactive Language Agents for Real-World Planning

A Human-Like Reasoning Framework for Multi-Phases Planning Task with Large Language Models

AgentBoard: An Analytical Evaluation Board of Multi-turn LLM Agents

Perceive, Reflect, and Plan: Designing LLM Agent for Goal-Directed City Navigation without Instructions

ScienceAgentBench: Toward Rigorous Assessment of Language Agents for Data-Driven Scientific Discovery

TDAG: A Multi-Agent Framework based on Dynamic Task Decomposition and Agent Generation

AgentBench: Evaluating LLMs as Agents