Fully Aligned Network for Referring Image Segmentation

Yong Liu,Ruihao Xu,Yansong Tang

2024-09-29

Abstract:This paper focuses on the Referring Image Segmentation (RIS) task, which aims to segment objects from an image based on a given language description. The critical problem of RIS is achieving fine-grained alignment between different modalities to recognize and segment the target object. Recent advances using the attention mechanism for cross-modal interaction have achieved excellent progress. However, current methods tend to lack explicit principles of interaction design as guidelines, leading to inadequate cross-modal comprehension. Additionally, most previous works use a single-modal mask decoder for prediction, losing the advantage of full cross-modal alignment. To address these challenges, we present a Fully Aligned Network (FAN) that follows four cross-modal interaction principles. Under the guidance of reasonable rules, our FAN achieves state-of-the-art performance on the prevalent RIS benchmarks (RefCOCO, RefCOCO+, G-Ref) with a simple architecture.

Computer Vision and Pattern Recognition

What problem does this paper attempt to address?

This paper attempts to solve the key problems in the Referring Image Segmentation (RIS) task, that is, to achieve fine - grained alignment between different modalities (vision and language) in order to accurately identify and segment target objects. Specifically, the paper mainly focuses on the following two challenges:

1. **Lack of clear interaction design principles**: Current methods lack clear principle guidance when designing cross - modal interactions, resulting in insufficient cross - modal understanding. This makes the model perform poorly in complex scenes.
2. **Limitations of single - modality decoders**: Most existing methods use single - modality mask decoders for prediction, failing to fully utilize the advantages of complete cross - modal alignment, thus losing the synergy of visual and language information.

To solve these problems, the authors propose a brand - new framework - **Fully Aligned Network (FAN)**, and summarize four cross - modal interaction principles to guide model design. These principles include:

- **Encoding interaction**: Initially activate visual features to reduce the influence of background pixels.
- **Coarse - and - fine - grained interaction**: Utilize word - level and sentence - level features to highlight target objects in detail.
- **Multi - scale interaction**: Utilize visual feature information at different levels.
- **Bidirectional interaction**: Update visual and language features simultaneously to create a joint space.

By following these principles, FAN achieves state - of - the - art performance on popular RIS benchmark datasets such as RefCOCO, RefCOCO +, and G - Ref.