IryoNLP at MEDIQA-CORR 2024: Tackling the Medical Error Detection & Correction Task On the Shoulders of Medical Agents

Jean-Philippe Corbeil

2024-04-24

Abstract:In natural language processing applied to the clinical domain, utilizing large language models has emerged as a promising avenue for error detection and correction on clinical notes, a knowledge-intensive task for which annotated data is scarce. This paper presents MedReAct'N'MedReFlex, which leverages a suite of four LLM-based medical agents. The MedReAct agent initiates the process by observing, analyzing, and taking action, generating trajectories to guide the search to target a potential error in the clinical notes. Subsequently, the MedEval agent employs five evaluators to assess the targeted error and the proposed correction. In cases where MedReAct's actions prove insufficient, the MedReFlex agent intervenes, engaging in reflective analysis and proposing alternative strategies. Finally, the MedFinalParser agent formats the final output, preserving the original style while ensuring the integrity of the error correction process. One core component of our method is our RAG pipeline based on our ClinicalCorp corpora. Among other well-known sources containing clinical guidelines and information, we preprocess and release the open-source MedWiki dataset for clinical RAG application. Our results demonstrate the central role of our RAG approach with ClinicalCorp leveraged through the MedReAct'N'MedReFlex framework. It achieved the ninth rank on the MEDIQA-CORR 2024 final leaderboard.

Computation and Language,Artificial Intelligence,Multiagent Systems

What problem does this paper attempt to address?

The paper aims to address the issue of medical error detection and correction in clinical notes. Specifically, the authors propose a multi-agent framework named MedReAct’N’MedReFlex to participate in the MEDIQA-CORR 2024 competition. This framework includes four specialized medical agents: MedReAct, MedReFlex, MedEval, and MedFinalParser. These agents work collaboratively to identify and correct potential errors in clinical notes through a Retrieval-Augmented Generation (RAG) framework.

The main contributions include:
1. Designing a multi-agent framework based on four medical agents, MedReAct’N’MedReFlex, to tackle the task of medical error detection and correction in the MEDIQA-CORR 2024 competition.
2. Releasing the open-source dataset MedWiki, which contains approximately 1.3 million article fragments focused on the medical field.
3. Providing a method to build a large corpus, ClinicalCorp, for RAG applications, which includes over 2.3 million document fragments.
4. Releasing a RAG version of the pre-training guide, containing over 710,000 fragments from eight open-source datasets.
5. Publishing the codebase on GitHub.

Through a series of experiments, the authors found that the optimal settings were to retrieve the top 50 documents, re-rank the top 20 documents, and set an evaluation threshold. These optimizations significantly improved the system's performance. Ultimately, the framework achieved 9th place in the MEDIQA-CORR 2024 competition.