Abstract:The current state-of-the-art image-sentence retrieval methods implicitly align the visual-textual fragments, like regions in images and words in sentences, and adopt attention modules to highlight the relevance of cross-modal semantic correspondences. However, the retrieval performance remains unsatisfactory due to a lack of consistent representation in both semantics and structural spaces. In this work, we propose to address the above issue from two aspects: (i) constructing intrinsic structure (along with relations) among the fragments of respective modalities, e.g., "dog $\to$ play $\to$ ball" in semantic structure for an image, and (ii) seeking explicit inter-modal structural and semantic correspondence between the visual and textual modalities. In this paper, we propose a novel Structured Multi-modal Feature Embedding and Alignment (SMFEA) model for image-sentence retrieval. In order to jointly and explicitly learn the visual-textual embedding and the cross-modal alignment, SMFEA creates a novel multi-modal structured module with a shared context-aware referral tree. In particular, the relations of the visual and textual fragments are modeled by constructing Visual Context-aware Structured Tree encoder (VCS-Tree) and Textual Context-aware Structured Tree encoder (TCS-Tree) with shared labels, from which visual and textual features can be jointly learned and optimized. We utilize the multi-modal tree structure to explicitly align the heterogeneous image-sentence data by maximizing the semantic and structural similarity between corresponding inter-modal tree nodes. Extensive experiments on Microsoft COCO and Flickr30K benchmarks demonstrate the superiority of the proposed model in comparison to the state-of-the-art methods.

Structural Embedding of Syntactic Trees for Machine Comprehension

When Are Tree Structures Necessary for Deep Learning of Representations?

Learning to Embed Sentences Using Attentive Recursive Trees.

Going "Deeper": Structured Sememe Prediction via Transformer with Tree Attention

Modelling Sentence Pairs with Tree-structured Attentive Encoder

Tree-Structured Neural Machine for Linguistics-Aware Sentence Generation

On Tree-Based Neural Sentence Modeling

How Well Do Text Embedding Models Understand Syntax?

Learning Better Sentence Representation with Syntax Information

Graph-to-Tree Neural Networks for Learning Structured Input-Output Translation with Applications to Semantic Parsing and Math Word Problem.

Incorporating Complete Syntactical Knowledge for Spoken Language Understanding

Structured Multi-modal Feature Embedding and Alignment for Image-Sentence Retrieval

S-Net: From Answer Extraction to Answer Synthesis for Machine Reading Comprehension

A Structured Self-attentive Sentence Embedding.

Improved Neural Machine Translation with a Syntax-Aware Encoder and Decoder

Bridging the Gap Between Synthetic and Natural Questions Via Sentence Decomposition for Semantic Parsing

Image-To-Tree: A Tree-Structured Decoder For Image Captioning

Dependency-based Convolutional Neural Networks for Sentence Embedding

Multimodal Deep Embedding via Hierarchical Grounded Compositional Semantics.

Improve Discourse Dependency Parsing with Contextualized Representations

A neural machine translation method based on split graph convolutional self-attention encoding