GeneGPT: augmenting large language models with domain tools for improved access to biomedical information

Qiao Jin,Yifan Yang,Qingyu Chen,Zhiyong Lu

DOI: https://doi.org/10.1093/bioinformatics/btae075

IF: 5.8

2024-02-01

Bioinformatics

Abstract:Abstract Motivation While large language models (LLMs) have been successfully applied to various tasks, they still face challenges with hallucinations. Augmenting LLMs with domain-specific tools such as database utilities can facilitate easier and more precise access to specialized knowledge. In this article, we present GeneGPT, a novel method for teaching LLMs to use the Web APIs of the National Center for Biotechnology Information (NCBI) for answering genomics questions. Specifically, we prompt Codex to solve the GeneTuring tests with NCBI Web APIs by in-context learning and an augmented decoding algorithm that can detect and execute API calls. Results Experimental results show that GeneGPT achieves state-of-the-art performance on eight tasks in the GeneTuring benchmark with an average score of 0.83, largely surpassing retrieval-augmented LLMs such as the new Bing (0.44), biomedical LLMs such as BioMedLM (0.08) and BioGPT (0.04), as well as GPT-3 (0.16) and ChatGPT (0.12). Our further analyses suggest that: First, API demonstrations have good cross-task generalizability and are more useful than documentations for in-context learning; second, GeneGPT can generalize to longer chains of API calls and answer multi-hop questions in GeneHop, a novel dataset introduced in this work; finally, different types of errors are enriched in different tasks, providing valuable insights for future improvements. Availability and implementation The GeneGPT code and data are publicly available at https://github.com/ncbi/GeneGPT.

biochemical research methods,biotechnology & applied microbiology,mathematical & computational biology

What problem does this paper attempt to address?

### Problems the Paper Attempts to Solve

The paper aims to address the issue of hallucinations in large language models (LLMs) when generating information, specifically the generation of content that appears plausible but is actually incorrect. Specifically, the authors propose a new method—GeneGPT—by teaching LLMs how to use the National Center for Biotechnology Information (NCBI) Web API to answer genomics-related questions. The main objectives include:

1. **Enhancing LLM Capabilities**: By integrating the NCBI Web API tool, enabling LLMs to access biomedical information more easily and accurately.
2. **Improving Question-Answering Performance**: In the GeneTuring benchmark test, GeneGPT significantly outperformed existing LLM models such as the new Bing, BioMedLM, BioGPT, etc.
3. **Multi-Step Reasoning Ability**: By introducing the GeneHop dataset, validating GeneGPT's ability to handle multi-step reasoning problems.

In summary, the paper primarily focuses on how to enhance LLMs' question-answering performance in the biomedical field and reduce the likelihood of generating incorrect information by integrating external tools like the NCBI Web API.