Towards Fast and Accurate Image-Text Retrieval with Self-Supervised Fine-Grained Alignment

Jiamin Zhuang,Jing Yu,Yang Ding,Xiangyan Qu,Yue Hu

DOI: https://doi.org/10.1109/TMM.2023.3280734

2023-08-27

Abstract:Image-text retrieval requires the system to bridge the heterogenous gap between vision and language for accurate retrieval while keeping the network lightweight-enough for efficient retrieval. Existing trade-off solutions mainly study from the view of incorporating cross-modal interactions with the independent-embedding framework or leveraging stronger pretrained encoders, which still demand time-consuming similarity measurement or heavyweight model structure in the retrieval stage. In this work, we propose an image-text alignment module SelfAlign on top of the independent-embedding framework, which improves the retrieval accuracy while maintains the retrieval efficiency without extra supervision. SelfAlign contains two collaborative sub-modules that force image-text alignment at both concept level and context level by self-supervised contrastive learning. It does not require cross-modal embedding interactions during training while maintaining independent image and text encoders during retrieval. With comparable time cost, SelfAlign consistently boosts the accuracy of state-of-the-art non-pretraining independent-embedding models respectively by 9.1%, 4.2% and 6.6% in terms of R@sum score on Flickr30K, MSCOCO 1K and MS-COCO 5K datasets. The retrieval accuracy also outperforms most existing interactive-embedding models with orders of magnitude decrease in retrieval time. The source code is available at: <a class="link-external link-https" href="https://github.com/Zjamie813/SelfAlign" rel="external noopener nofollow">this https URL</a>.

Computer Vision and Pattern Recognition,Artificial Intelligence

What problem does this paper attempt to address?

The problem that this paper attempts to solve is how to improve the retrieval accuracy while maintaining high efficiency in the image - text retrieval task. Specifically, the existing methods either focus on retrieval efficiency, such as independent embedding models. Although these models can achieve fast retrieval, they cannot provide fine - grained content alignment, resulting in low retrieval precision; or they focus on retrieval accuracy, such as interactive embedding models. These models achieve fine - grained image - text matching through cross - modal attention mechanisms, but they have high computational complexity and are not suitable for large - scale online retrieval scenarios.

The paper proposes a new method, called SelfAlign, which aims to learn fine - grained image - text alignment through self - supervised contrastive learning while maintaining the efficiency of the independent embedding model. SelfAlign contains two sub - modules:

1. **Local Concept Alignment (LCA)**: This sub - module is injected at the object and word encoding layers, aiming to enforce consistency between visual and textual concept embeddings. It achieves this by discovering pseudo - word - object correspondences and using a clustering - based fine - grained alignment strategy.

2. **Contextual Relation Alignment (CRA)**: This sub - module is injected at the context encoding layer, aiming to capture semantic correspondences at the context level. It first performs shared context enhancement and then context - level alignment.

Through these two sub - modules, SelfAlign can significantly improve the retrieval precision of the independent embedding model without sacrificing efficiency. The experimental results show that SelfAlign improves the R@sum scores by 9.1%, 4.2% and 6.6% respectively on multiple datasets, while the retrieval time is much lower than that of most existing interactive embedding models.