Fine-Tuning Medical Language Models for Enhanced Long-Contextual Understanding and Domain Expertise

Qimin Yang,Rongsheng Wang,Jiexin Chen,Runqi Su,Tao Tan

2024-07-16

Abstract:Large Language Models (LLMs) have been widely applied in various professional fields. By fine-tuning the models using domain specific question and answer datasets, the professional domain knowledge and Q\&A abilities of these models have significantly improved, for example, medical professional LLMs that use fine-tuning of doctor-patient Q\&A data exhibit extraordinary disease diagnostic abilities. However, we observed that despite improvements in specific domain knowledge, the performance of medical LLM in long-context understanding has significantly declined, especially compared to general language models with similar parameters. The purpose of this study is to investigate the phenomenon of reduced performance in understanding long-context in medical LLM. We designed a series of experiments to conduct open-book professional knowledge exams on all models to evaluate their ability to read long-context. By adjusting the proportion and quantity of general data and medical data in the process of fine-tuning, we can determine the best data composition to optimize the professional model and achieve a balance between long-context performance and specific domain knowledge.

Computation and Language,Artificial Intelligence

What problem does this paper attempt to address?

The paper aims to address the issue of large language models in the medical field (Medical Large Language Models, Medical LLMs) lacking the ability to understand long texts. Although these models have significantly improved their knowledge and question-answering abilities in specialized fields through fine-tuning with domain-specific data, their capabilities in handling long text comprehension and instruction execution have declined. Specifically, in multi-turn dialogues or situations requiring long-term contextual information, the models perform worse than general language models.

To explore the reasons behind this phenomenon and find solutions, the researchers designed a series of experiments. They fine-tuned the models by adjusting the ratio and quantity of general data to medical data to determine the optimal data combination strategy, thereby optimizing the specialized models to balance long text comprehension and domain-specific knowledge. The experimental results show that increasing a certain proportion of general data can effectively enhance the long text comprehension ability of medical LLMs, and the impact of different data quantities on model performance also varies. Overall, the goal of the paper is to find a method that allows medical LLMs to maintain specialized knowledge while also possessing good broad language understanding capabilities.