Eye-gaze Guided Multi-modal Alignment for Medical Representation Learning

Chong Ma,Hanqi Jiang,Wenting Chen,Yiwei Li,Zihao Wu,Xiaowei Yu,Zhengliang Liu,Lei Guo,Dajiang Zhu,Tuo Zhang,Dinggang Shen,Tianming Liu,Xiang Li

2024-06-14

Abstract:In the medical multi-modal frameworks, the alignment of cross-modality features presents a significant challenge. However, existing works have learned features that are implicitly aligned from the data, without considering the explicit relationships in the medical context. This data-reliance may lead to low generalization of the learned alignment relationships. In this work, we propose the Eye-gaze Guided Multi-modal Alignment (EGMA) framework to harness eye-gaze data for better alignment of medical visual and textual features. We explore the natural auxiliary role of radiologists' eye-gaze data in aligning medical images and text, and introduce a novel approach by using eye-gaze data, collected synchronously by radiologists during diagnostic evaluations. We conduct downstream tasks of image classification and image-text retrieval on four medical datasets, where EGMA achieved state-of-the-art performance and stronger generalization across different datasets. Additionally, we explore the impact of varying amounts of eye-gaze data on model performance, highlighting the feasibility and utility of integrating this auxiliary data into multi-modal alignment framework.

Computer Vision and Pattern Recognition,Computation and Language

What problem does this paper attempt to address?

This paper attempts to address the challenge of cross-modal feature alignment in the medical multimodal framework. Existing works, although capable of implicitly learning aligned features from data, do not consider explicit relationships in the medical context, which may lead to low generalization ability of the learned alignment relationships. Therefore, this paper proposes an Eye Gaze-guided Multimodal Alignment (EGMA) framework, which utilizes eye movement data synchronously collected from radiologists during diagnosis to better align medical visual and textual features.

Specifically, the paper mainly addresses the following issues:
1. **Challenge of cross-modal feature alignment**: Existing methods rely on the data itself when aligning image and text features, without fully utilizing explicit relationships in the medical context, leading to insufficient generalization ability.
2. **Data scarcity issue**: In the medical field, high-quality paired image and text data are relatively scarce, making it difficult for models to learn effective cross-modal feature alignment.
3. **Improving model performance**: By introducing eye movement data from radiologists, the model's performance in downstream tasks such as image classification and image-text retrieval is enhanced, improving the model's generalization ability and robustness.

To validate the effectiveness of the proposed EGMA framework, the authors conducted experiments on 4 medical datasets. The results show that EGMA achieved state-of-the-art performance in zero-shot classification and retrieval tasks and demonstrated stronger generalization ability under different data scales. Additionally, the study explored the impact of different amounts of eye movement data on model performance, finding that even a small amount of eye movement data can significantly enhance the model's multimodal processing capability.