BioMamba: A Pre-trained Biomedical Language Representation Model Leveraging Mamba

Ling Yue,Sixue Xing,Yingzhou Lu,Tianfan Fu

2024-08-06

Abstract:The advancement of natural language processing (NLP) in biology hinges on models' ability to interpret intricate biomedical literature. Traditional models often struggle with the complex and domain-specific language in this field. In this paper, we present BioMamba, a pre-trained model specifically designed for biomedical text mining. BioMamba builds upon the Mamba architecture and is pre-trained on an extensive corpus of biomedical literature. Our empirical studies demonstrate that BioMamba significantly outperforms models like BioBERT and general-domain Mamba across various biomedical tasks. For instance, BioMamba achieves a 100 times reduction in perplexity and a 4 times reduction in cross-entropy loss on the BioASQ test set. We provide an overview of the model architecture, pre-training process, and fine-tuning techniques. Additionally, we release the code and trained model to facilitate further research.

Computation and Language

What problem does this paper attempt to address?

The paper introduces BioMamba, a pre-trained language model specifically designed for biomedical text mining. The primary goal of this work is to address the challenges associated with interpreting complex and domain-specific language found in biomedical literature, which traditional models often struggle with.

Key points addressed in the paper include:

1. **Problem Statement**: There is a need for advanced models that can effectively handle biomedical text mining tasks, considering the exponential growth in biomedical literature and the complexity of the language used.

2. **Solution Proposed**: The authors propose BioMamba, a domain-specific adaptation of the Mamba model. BioMamba is initialized with the Mamba model and then fine-tuned for biomedical text mining. It leverages the Mamba architecture, which is based on structured state space models (SSMs) with parameters that are functions of the input, offering linear complexity in sequence length and efficient handling of long sequences.

3. **Contributions**:
- Development of BioMamba, which is pre-trained on PubMed abstracts and fine-tuned for specific biomedical tasks.
- Empirical evaluation demonstrating BioMamba's superiority over existing methods such as BioBERT, BioGPT, and the general-purpose Mamba model.
- Public release of the well-trained BioMamba model to facilitate biomedical research.

4. **Performance Evaluation**:
- BioMamba shows significant improvements over existing methods in various biomed