Segmentation-guided Attention for Visual Question Answering from Remote Sensing Images

Lucrezia Tosato,Hichem Boussaid,Flora Weissgerber,Camille Kurtz,Laurent Wendling,Sylvain Lobry

2024-07-12

Abstract:Visual Question Answering for Remote Sensing (RSVQA) is a task that aims at answering natural language questions about the content of a remote sensing image. The visual features extraction is therefore an essential step in a VQA pipeline. By incorporating attention mechanisms into this process, models gain the ability to focus selectively on salient regions of the image, prioritizing the most relevant visual information for a given question. In this work, we propose to embed an attention mechanism guided by segmentation into a RSVQA pipeline. We argue that segmentation plays a crucial role in guiding attention by providing a contextual understanding of the visual information, underlying specific objects or areas of interest. To evaluate this methodology, we provide a new VQA dataset that exploits very high-resolution RGB orthophotos annotated with 16 segmentation classes and question/answer pairs. Our study shows promising results of our new methodology, gaining almost 10% of overall accuracy compared to a classical method on the proposed dataset.

Computer Vision and Pattern Recognition

What problem does this paper attempt to address?

The problem that this paper attempts to solve is to improve the accuracy of the remote - sensing image visual question - answering (RSVQA) task. Specifically, the author proposes a method based on the segmentation - guided attention mechanism to better handle the visual information in remote - sensing images and align it with natural - language questions. Through this method, the model can focus more precisely on the regions in the image related to the questions, thereby improving the accuracy of the answers.

### Problem Background

Remote - sensing image visual question - answering (RSVQA) is a task aimed at answering natural - language questions based on the content of remote - sensing images. In this process, extracting effective visual features is a very important step. Traditional VQA methods usually use global or local visual features, but these methods may not be able to effectively capture the information of specific regions in the image, especially in complex remote - sensing images.

### Core Contributions of the Paper

1. **Segmentation - guided Attention Mechanism**: The author proposes a new method, introducing semantic segmentation into the attention mechanism to guide the model to focus on the key regions in the image. In this way, the model can better understand the objects in the image and their contextual relationships, thereby improving the accuracy of the answers.

2. **New Dataset**: To evaluate the effectiveness of this method, the author constructs a new RSVQA dataset, which contains very high - resolution RGB orthophotos, 16 - class semantic segmentation labels, and automatically generated question/answer pairs. This dataset covers four adjacent provinces in the greater Paris region of France, providing rich geographical information and diverse scenes.

3. **Experimental Verification**: The author verifies the effectiveness of the segmentation - guided attention mechanism through experiments. The experimental results show that, compared with traditional VQA methods, this method improves the overall accuracy by nearly 10% and shows better performance in different types of question - answering tasks.

### Formula Explanation

In the paper, the author does not involve a large number of complex formulas, but mentions some key operations when describing the model structure:

- **Attention Weight Calculation**:
\[
a=\text{ReLU}(\text{Conv}(f_{\text{seg}} + f_q))
\]
where \( f_{\text{seg}} \) is the feature map obtained from the segmentation module, \( f_q \) is the feature extracted from the question text, \( \text{Conv} \) represents the convolution operation, and \( \text{ReLU} \) is the activation function.

- **Prediction Output**:
\[
y = \text{MLP}(\text{Concat}(a\cdot f_{\text{VHR}}, f_q))
\]
where \( \text{MLP} \) is a multi - layer perceptron used to map the fused features to the final output space, and \( \text{Concat} \) represents the feature concatenation operation.

### Summary

The main goal of this paper is to improve the performance of the remote - sensing image visual question - answering task by introducing the segmentation - guided attention mechanism. The experimental results show that this method can significantly improve the accuracy of the model and provide new ideas and directions for future research.