MolCap-Arena: A Comprehensive Captioning Benchmark on Language-Enhanced Molecular Property Prediction

Carl Edwards,Ziqing Lu,Ehsan Hajiramezanali,Tommaso Biancalani,Heng Ji,Gabriele Scalia

2024-11-02

Abstract:Bridging biomolecular modeling with natural language information, particularly through large language models (LLMs), has recently emerged as a promising interdisciplinary research area. LLMs, having been trained on large corpora of scientific documents, demonstrate significant potential in understanding and reasoning about biomolecules by providing enriched contextual and domain knowledge. However, the extent to which LLM-driven insights can improve performance on complex predictive tasks (e.g., toxicity) remains unclear. Further, the extent to which relevant knowledge can be extracted from LLMs also remains unknown. In this study, we present Molecule Caption Arena: the first comprehensive benchmark of LLM-augmented molecular property prediction. We evaluate over twenty LLMs, including both general-purpose and domain-specific molecule captioners, across diverse prediction tasks. To this goal, we introduce a novel, battle-based rating system. Our findings confirm the ability of LLM-extracted knowledge to enhance state-of-the-art molecular representations, with notable model-, prompt-, and dataset-specific variations. Code, resources, and data are available at <a class="link-external link-http" href="http://github.com/Genentech/molcap-arena" rel="external noopener nofollow">this http URL</a>.

Computation and Language,Artificial Intelligence,Biomolecules

What problem does this paper attempt to address?

The main problem this paper attempts to address is the evaluation of the usefulness of molecular descriptions (i.e., molecular captions) generated by large language models (LLMs) in real-world biomolecular property prediction tasks. Specifically, the authors propose a benchmark platform named MolCap-Arena, which aims to:

1. **Establish a large-scale molecular caption generator benchmark**: Going beyond traditional n-gram-based evaluation metrics, focusing on real-world tasks such as toxicity prediction.
2. **Provide molecular captions and a unified codebase**: Including dozens of models to facilitate comparisons for future researchers.
3. **Analyze the capabilities of existing models**: Through hundreds of experiments, investigating whether molecular captions can capture relevant biochemical knowledge.

Through these objectives, the paper hopes to improve existing molecular property prediction models, enhancing their interpretability and accuracy in practical applications.