Abstract:The advancements in computer vision and natural language processing are keys to thriving modern healthcare systems and its applications. Nonetheless, they have been researched and used as separate technical entities without integrating their predictive knowledge discovery when they are combined. Such integration will benefit every clinical/medical problem as they are inherently multimodal - they involve several distinct forms of data, such as images and text. However, the recent advancements in machine learning have brought these fields closer using the notion of meta-transformers. At the core of this synergy is building models that can process and relate information from multiple modalities where the raw input data from various modalities are mapped into a shared token space, allowing an encoder to extract high-level semantic features of the input data. Nerveless, the task of automatically identifying arguments in a clinical/medical text and finding their multimodal relationships remains challenging as it does not rely only on relevancy measures (e.g. how close that text to other modalities like an image) but also on the evidence supporting that relevancy. Relevancy based on evidence is a normal practice in medicine as every practice is an evidence-based. In this article we are experimenting with meta-transformers that can benefit evidence based predictions. In this article, we are experimenting with variety of fine tuned medical meta-transformers like PubmedCLIP, CLIPMD, BiomedCLIP-PubMedBERT and BioCLIP to see which one provide evidence-based relevant multimodal information. Our experimentation uses the TTi-Eval open-source platform to accommodate multimodal data embeddings. This platform simplifies the integration and evaluation of different meta-transformers models but also to variety of datasets for testing and fine tuning. Additionally, we are conducting experiments to test how relevant any multimodal prediction to the published medical literature especially those that are published by PubMed. Our experimentations revealed that the BiomedCLIP-PubMedBERT model provide more reliable evidence-based relevance compared to other models based on randomized samples from the ROCO V2 dataset or other multimodal datasets like MedCat. In this next stage of this research we are extending the use of the winning evidence-based multimodal learning model by adding components that enable medical practitioner to use this model to predict answers to clinical questions based on sound medical questioning protocol like PICO and based on standardized medical terminologies like UMLS.

Multi-Modal Perceiver Language Model for Outcome Prediction in Emergency Department

Multimodal Data Matters: Language Model Pre-Training Over Structured and Unstructured Electronic Health Records

Multimodal Clinical Pseudo-notes for Emergency Department Prediction Tasks using Multiple Embedding Model for EHR (MEME)

Multimodal risk prediction with physiological signals, medical images and clinical notes

Multimodal Large Language Models in Health Care: Applications, Challenges, and Future Outlook

Multimodal Data Hybrid Fusion and Natural Language Processing for Clinical Prediction Models

Med-PMC: Medical Personalized Multi-modal Consultation with a Proactive Ask-First-Observe-Next Paradigm

Improving Clinical Outcome Predictions Using Convolution over Medical Entities with Multimodal Learning

Learning Missing Modal Electronic Health Records with Unified Multi-modal Data Embedding and Modality-Aware Attention

How to Leverage Multimodal EHR Data for Better Medical Predictions?

MoRE: Multi-Modal Contrastive Pre-training with Transformers on X-Rays, ECGs, and Diagnostic Report

Two heads are better than one: Enhancing medical representations by pre-training over structured and unstructured electronic health records

Improving Early Sepsis Prediction with Multi Modal Learning

Using Meta-Transformers for Multimodal Clinical Decision Support and Evidence-Based Medicine

EMERGE: Enhancing Multimodal Electronic Health Records Predictive Modeling with Retrieval-Augmented Generation

MDA: An Interpretable and Scalable Multi-Modal Fusion under Missing Modalities and Intrinsic Noise Conditions

M$^3$Care: Learning with Missing Modalities in Multimodal Healthcare Data

Emergency Department Decision Support using Clinical Pseudo-notes

REALM: RAG-Driven Enhancement of Multimodal Electronic Health Records Analysis via Large Language Models

Inter-modality Dependence Induced Data Recovery for MCI Conversion Prediction