Abstract:Background: Understanding complex biological pathways, including gene-gene interactions and gene regulatory networks, is critical for exploring disease mechanisms and drug development. Manual literature curation of biological pathways is useful but cannot keep up with the exponential growth of the literature. Large-scale language models (LLMs), notable for their vast parameter sizes and comprehensive training on extensive text corpora, have great potential in automated text mining of biological pathways. Method: This study assesses the effectiveness of 21 LLMs, including both API-based models and open-source models. The evaluation focused on two key aspects: gene regulatory relations (specifically, 'activation', 'inhibition', and 'phosphorylation') and KEGG pathway component recognition. The performance of these models was analyzed using statistical metrics such as precision, recall, F1 scores, and the Jaccard similarity index. Results: Our results indicated a significant disparity in model performance. Among the API-based models, ChatGPT-4 and Claude-Pro showed superior performance, with an F1 score of 0.4448 and 0.4386 for the gene regulatory relation prediction, and a Jaccard similarity index of 0.2778 and 0.2657 for the KEGG pathway prediction, respectively. Open-source models lagged their API-based counterparts, where Falcon-180b-chat and llama1-7b led with the highest performance in gene regulatory relations (F1 of 0.2787 and 0.1923, respectively) and KEGG pathway recognition (Jaccard similarity index of 0.2237 and 0. 2207, respectively). Conclusion: LLMs are valuable in biomedical research, especially in gene network analysis and pathway mapping. However, their effectiveness varies, necessitating careful model selection. This work also provided a case study and insight into using LLMs as knowledge graphs.

A machine learning framework for extracting information from biological pathway images in the literature

Web-based information retrieval system for the prediction of metabolic pathways.

PathwayFinder : Bridging the Way Towards Automatic Pathway Extraction

PathwayFinder: Paving the Way Towards Automatic Pathway Extraction.

pathCLIP: Detection of Genes and Gene Relations From Biological Pathway Figures Through Image-Text Contrastive Learning

MLGL-MP: a Multi-Label Graph Learning framework enhanced by pathway interdependence for Metabolic Pathway prediction

Automatic pathway information extraction from biomedical literature

A comprehensive evaluation of large language models in mining gene relations and pathway knowledge

A syntactic rule-based method for automatic pathway information extraction from biomédical literature

Comprehensive Analysis of Features and Annotations of Pathway Databases

A Knowledge Graph Approach to Elucidate the Role of Organellar Pathways in Disease via Biomedical Reports

PathNER: a tool for systematic identification of biological pathway mentions in the literature

A Comprehensive Evaluation of Large Language Models in Mining Gene Interactions and Pathway Knowledge

BioKEEN: a library for learning and evaluating biological knowledge graph embeddings

Biomedical Information Extraction for Disease Gene Prioritization

Chapter 14: Automated Mining of Disease-Specific Protein Interaction Networks Based on Biomedical Literature

A deep learning architecture for metabolic pathway prediction

Biological Network Extraction from Scientific Literature: State of the Art and Challenges

SubpathwayMiner: a Software Package for Flexible Identification of Pathways

External Evaluation of Event Extraction Classifiers for Automatic Pathway Curation: An extended study of the mTOR pathway

Automatically layout and visualize the biological pathway map with spectral graph theory