Learning Aligned Image-Text Representations Using Graph Attentive Relational Network

Ya Jing,Wei Wang,Liang Wang,Tieniu Tan

DOI: https://doi.org/10.1109/tip.2020.3048627

IF: 10.6

2021-01-01

IEEE Transactions on Image Processing

Abstract:Image-text matching aims to measure the similarities between images and textual descriptions, which has made great progress recently. The key to this cross-modal matching task is to build the latent semantic alignment between visual objects and words. Due to the widespread variations of sentence structures, it is very difficult to learn the latent semantic alignment using only global cross-modal features. Many previous methods attempt to learn the aligned image-text representations by the attention mechanism but generally ignore the relationships within textual descriptions which determine whether the words belong to the same visual object. In this paper, we propose a graph attentive relational network (GARN) to learn the aligned image-text representations by modeling the relationships between noun phrases in a text for the identity-aware image-text matching. In the GARN, we first decompose images and texts into regions and noun phrases, respectively. Then a skip graph neural network (skip-GNN) is proposed to learn effective textual representations which are a mixture of textual features and relational features. Finally, a graph attention network is further proposed to obtain the probabilities that the noun phrases belong to the image regions by modeling the relationships between noun phrases. We perform extensive experiments on the CUHK Person Description dataset (CUHK-PEDES), Caltech-UCSD Birds dataset (CUB), Oxford-102 Flowers dataset and Flickr30K dataset to verify the effectiveness of each component in our model. Experimental results show that our approach achieves the state-of-the-art results on these four benchmark datasets.

computer science, artificial intelligence,engineering, electrical & electronic

What problem does this paper attempt to address?

The problem that this paper attempts to solve is how to effectively establish the latent semantic alignment between images and text descriptions in cross - modal matching tasks, especially for identity - aware image - text matching tasks. Due to the wide variety of sentence structures, it is very difficult to directly use global cross - modal features to learn this latent semantic alignment. Many previous methods attempt to learn aligned image - text representations through the attention mechanism, but usually ignore the relationships within text descriptions, which determine whether words belong to the same visual object.

To solve the above problems, the authors propose a Graph Attentive Relational Network (GARN), which learns aligned image - text representations by modeling the relationships between noun phrases in the text. GARN first decomposes the image and text into regions and noun phrases, and then uses the skip - Graph Neural Network (skip - GNN) to learn effective text representations, which are a mixture of text features and relationship features. Finally, the probability that noun phrases belong to image regions is further learned through the graph attention network, thereby achieving identity - aware image - text matching.

The main contributions of the paper include:
1. Proposing a novel Graph Attentive Relational Network (GARN) to learn aligned image - text representations.
2. Designing a new skip - Graph Neural Network, aiming to learn more effective text representations by integrating text features and relationship features.
3. Explicitly modeling the relationships between noun phrases through a new graph attention network to learn latent visual - semantic alignment.
4. Achieving the best performance on four challenging benchmark datasets, verifying the effectiveness of the proposed model.

These contributions make GARN perform well in handling complex relationships and non - linear mappings, especially in identity - aware image - text matching tasks.