Abstract:There is widespread optimism that frontier Large Language Models (LLMs) and LLM-augmented systems have the potential to rapidly accelerate scientific discovery across disciplines. Today, many benchmarks exist to measure LLM knowledge and reasoning on textbook-style science questions, but few if any benchmarks are designed to evaluate language model performance on practical tasks required for scientific research, such as literature search, protocol planning, and data analysis. As a step toward building such benchmarks, we introduce the Language Agent Biology Benchmark (LAB-Bench), a broad dataset of over 2,400 multiple choice questions for evaluating AI systems on a range of practical biology research capabilities, including recall and reasoning over literature, interpretation of figures, access and navigation of databases, and comprehension and manipulation of DNA and protein sequences. Importantly, in contrast to previous scientific benchmarks, we expect that an AI system that can achieve consistently high scores on the more difficult LAB-Bench tasks would serve as a useful assistant for researchers in areas such as literature search and molecular cloning. As an initial assessment of the emergent scientific task capabilities of frontier language models, we measure performance of several against our benchmark and report results compared to human expert biology researchers. We will continue to update and expand LAB-Bench over time, and expect it to serve as a useful tool in the development of automated research systems going forward. A public subset of LAB-Bench is available for use at the following URL: <a class="link-external link-https" href="https://huggingface.co/datasets/futurehouse/lab-bench" rel="external noopener nofollow">this https URL</a>

Language Model Powered Digital Biology with BRAD

A Platform for the Biomedical Application of Large Language Models

Open‐source large language models in action: A bioinformatics chatbot for PRIDE database

BioInformatics Agent (BIA): Unleashing the Power of Large Language Models to Reshape Bioinformatics Workflow

ChatGPT, Bard, and Large Language Models for Biomedical Research: Opportunities and Pitfalls

LLaVA-Med: Training a Large Language-and-Vision Assistant for Biomedicine in One Day

LAB-Bench: Measuring Capabilities of Language Models for Biology Research

A language model assistant for biocatalysis

Bio-Eng-LMM AI Assist chatbot: A Comprehensive Tool for Research and Education

Chat with the Environment: Interactive Multimodal Perception Using Large Language Models

ChatLogo: A Large Language Model-Driven Hybrid Natural-Programming Language Interface for Agent-based Modeling and Programming

A Refer-and-Ground Multimodal Large Language Model for Biomedicine

Hello Again! LLM-powered Personalized Agent for Long-term Dialogue

Exploring Autonomous Agents through the Lens of Large Language Models: A Review

An Evaluation of Large Language Models in Bioinformatics Research

Advancing High Resolution Vision-Language Models in Biomedicine

ChatGPT for digital pathology research

AI for Biomedicine in the Era of Large Language Models

Domain-specific ChatBots for Science using Embeddings