Surgical-VQLA++: Adversarial Contrastive Learning for Calibrated Robust Visual Question-Localized Answering in Robotic Surgery

Long Bai,Guankun Wang,Mobarakol Islam,Lalithkumar Seenivasan,An Wang,Hongliang Ren

2024-09-01

Abstract:Medical visual question answering (VQA) bridges the gap between visual information and clinical decision-making, enabling doctors to extract understanding from clinical images and videos. In particular, surgical VQA can enhance the interpretation of surgical data, aiding in accurate diagnoses, effective education, and clinical interventions. However, the inability of VQA models to visually indicate the regions of interest corresponding to the given questions results in incomplete comprehension of the surgical scene. To tackle this, we propose the surgical visual question localized-answering (VQLA) for precise and context-aware responses to specific queries regarding surgical images. Furthermore, to address the strong demand for safety in surgical scenarios and potential corruptions in image acquisition and transmission, we propose a novel approach called Calibrated Co-Attention Gated Vision-Language (C$^2$G-ViL) embedding to integrate and align multimodal information effectively. Additionally, we leverage the adversarial sample-based contrastive learning strategy to boost our performance and robustness. We also extend our EndoVis-18-VQLA and EndoVis-17-VQLA datasets to broaden the scope and application of our data. Extensive experiments on the aforementioned datasets demonstrate the remarkable performance and robustness of our solution. Our solution can effectively combat real-world image corruption. Thus, our proposed approach can serve as an effective tool for assisting surgical education, patient care, and enhancing surgical outcomes.

Computer Vision and Pattern Recognition,Robotics

What problem does this paper attempt to address?

The paper aims to address the limitations of medical Visual Question Answering (VQA) in robotic surgery, particularly the issue where existing VQA models cannot clearly identify the regions of interest related to a given question. Specifically, the paper proposes the following points:

1. **Proposing the Surgical-VQLA++ Framework**: This framework combines answering and localization functions, significantly enhancing performance and robustness, making it more suitable for clinical applications. Additionally, the feature extraction strategy promotes global understanding, supports end-to-end solutions, and achieves efficient inference speed (150.6 FPS).

2. **Introducing the Calibrated Collaborative Attention Gated Vision-Language Embedding Module (C2G-ViL)**: This module optimizes the alignment and interaction of multimodal representations and calibrates contextual features globally. By exploring optimal fusion weights, this module aids in the alignment between different modalities.

3. **Adopting an Adversarial Contrastive Training Strategy**: Specific adversarial contrastive training based on the DeiT backbone network is employed to enhance the model's ability to capture subtle feature perturbations, thereby further improving performance and robustness. Various combinations of loss functions are explored to achieve multi-task convergence.

4. **Expanding the Dataset**: Based on the public EndoVis18 and EndoVis17 datasets, two comprehensive surgical datasets are expanded, containing a total of 17,269 question-answer pairs covering surgical organs, instruments, actions, and instrument positions, along with corresponding bounding boxes for answer localization.

Through these improvements, the method can effectively handle real-world image noise in cases of image quality degradation, thereby assisting in surgical education, patient care, and improving surgical outcomes.