Abstract:There is widespread optimism that frontier Large Language Models (LLMs) and LLM-augmented systems have the potential to rapidly accelerate scientific discovery across disciplines. Today, many benchmarks exist to measure LLM knowledge and reasoning on textbook-style science questions, but few if any benchmarks are designed to evaluate language model performance on practical tasks required for scientific research, such as literature search, protocol planning, and data analysis. As a step toward building such benchmarks, we introduce the Language Agent Biology Benchmark (LAB-Bench), a broad dataset of over 2,400 multiple choice questions for evaluating AI systems on a range of practical biology research capabilities, including recall and reasoning over literature, interpretation of figures, access and navigation of databases, and comprehension and manipulation of DNA and protein sequences. Importantly, in contrast to previous scientific benchmarks, we expect that an AI system that can achieve consistently high scores on the more difficult LAB-Bench tasks would serve as a useful assistant for researchers in areas such as literature search and molecular cloning. As an initial assessment of the emergent scientific task capabilities of frontier language models, we measure performance of several against our benchmark and report results compared to human expert biology researchers. We will continue to update and expand LAB-Bench over time, and expect it to serve as a useful tool in the development of automated research systems going forward. A public subset of LAB-Bench is available for use at the following URL: <a class="link-external link-https" href="https://huggingface.co/datasets/futurehouse/lab-bench" rel="external noopener nofollow">this https URL</a>

A language model assistant for biocatalysis

Integrating Genetic Algorithms and Language Models for Enhanced Enzyme Design

Language models in molecular discovery

Synergizing Human Expertise and AI Efficiency with Language Model for Microscopy Operation and Automated Experiment Design

Large Language Models for Biomolecular Analysis: from Methods to Applications

Conditional language models enable the efficient design of proficient enzymes

SynAsk: Unleashing the Power of Large Language Models in Organic Synthesis

LAB-Bench: Measuring Capabilities of Language Models for Biology Research

Language Model Powered Digital Biology with BRAD

Automation and Machine Learning Augmented by Large Language Models in Catalysis Study

InstructBioMol: Advancing Biomolecule Understanding and Design Following Human Instructions

Language models can identify enzymatic binding sites in protein sequences

Utilizing Large Language Models for Natural Interface to Pharmacology Databases

Identification of Enzymatic Active Sites with Unsupervised Language Modeling

A Platform for the Biomedical Application of Large Language Models

A Review of Large Language Models and Autonomous Agents in Chemistry

Accelerated end-to-end chemical synthesis development with large language models

BioinspiredLLM: Conversational Large Language Model for the Mechanics of Biological and Bio-inspired Materials

Advancing Plant Metabolic Research By Using Large Language Models To Expand Databases And Extract Labelled Data

BioInformatics Agent (BIA): Unleashing the Power of Large Language Models to Reshape Bioinformatics Workflow

Fluent Integration of Laboratory Data into Biocatalytic Process Simulation Using EnzymeML, DWSIM, and Ontologies