Enhancing Adverse Drug Event Detection with Multimodal Dataset: Corpus Creation and Model Development

Pranab Sahoo,Ayush Kumar Singh,Sriparna Saha,Aman Chadha,Samrat Mondal

2024-05-25

Abstract:The mining of adverse drug events (ADEs) is pivotal in pharmacovigilance, enhancing patient safety by identifying potential risks associated with medications, facilitating early detection of adverse events, and guiding regulatory decision-making. Traditional ADE detection methods are reliable but slow, not easily adaptable to large-scale operations, and offer limited information. With the exponential increase in data sources like social media content, biomedical literature, and Electronic Medical Records (EMR), extracting relevant ADE-related information from these unstructured texts is imperative. Previous ADE mining studies have focused on text-based methodologies, overlooking visual cues, limiting contextual comprehension, and hindering accurate interpretation. To address this gap, we present a MultiModal Adverse Drug Event (MMADE) detection dataset, merging ADE-related textual information with visual aids. Additionally, we introduce a framework that leverages the capabilities of LLMs and VLMs for ADE detection by generating detailed descriptions of medical images depicting ADEs, aiding healthcare professionals in visually identifying adverse events. Using our MMADE dataset, we showcase the significance of integrating visual cues from images to enhance overall performance. This approach holds promise for patient safety, ADE awareness, and healthcare accessibility, paving the way for further exploration in personalized healthcare.

Artificial Intelligence,Computation and Language,Computer Vision and Pattern Recognition

What problem does this paper attempt to address?

This paper aims to address the limitations of adverse drug event (ADE) detection in pharmacovigilance. Traditional methods are reliable but slow, not suitable for large-scale operations, and limited in information. With the explosive growth of data sources such as social media, biomedical literature, and electronic medical records (EMR), it is crucial to extract relevant information related to ADEs from these unstructured texts. Although previous research has focused mainly on text-based methods, they have overlooked visual clues, limiting contextual understanding and affecting accurate interpretation.

To address this, the paper proposes a multimodal ADE detection dataset (MMADE) that combines text information related to ADEs with visual assistance. Additionally, they introduce a framework that utilizes large-scale language models (LLMs) and visual language models (VLMs) to generate detailed descriptions of medical images to help healthcare professionals identify adverse events through visual recognition, improving diagnostic accuracy and patient care quality. Through the MMADE dataset, the study demonstrates that incorporating visual clues from images can significantly enhance overall performance.

The paper also points out that although LLMs and VLMs have shown remarkable performance in certain medical applications, their potential in ADE detection has not been fully explored because they are primarily trained on generic natural images and may lack the specialized knowledge required to handle complex medical images. Therefore, they fine-tune these models to adapt to the specific requirements of ADE tasks.

In summary, the main contributions of the paper include:
1. Proposing a multimodal approach to enhance ADE detection, assisting doctors, nurses, and pharmacists in providing detailed descriptions of ADE cases and improving diagnostic accuracy.
2. Introducing a new multimodal dataset, MMADE, for further ADE detection research.
3. Demonstrating the potential applications of MMADE in ADE classification, caption generation, and summary tasks.
4. Experimenting and analyzing fine-tuned InstructBLIP and other pre-trained VLMs.

In this way, the paper opens up new avenues for improving patient safety, increasing ADE awareness, and enhancing medical communication, especially for individuals who need to understand potential drug risk information accompanied by medical images.