Abstract:Bridging visual and textual representations plays a central role in delving into multimedia data understanding. The main challenge arises from that images and texts exist in heterogeneous spaces, leading to the difficulty to preserve the semantic consistency between both modalities. To narrow the modality gap, most recent methods resort to extra object detectors or parsers to obtain the hierarchical representations. In this work, we address this problem by introducing our Multi-Task Hierarchical Convolutional Neural Network (MT-HCN). It is characterized by mining the hierarchical semantic information without the aid of any extra supervisions. Firstly, from the perspective of representing architecture, we leverage the intrinsic hierarchical structure of Convolutional Neural Networks (CNNs) to decompose the representations of both modalities into two semantically complementary levels, i.e. , exterior representations and concept representations. The former focuses on discovering the fine-grained low-level associations between both modalities, meanwhile the latter underlines capturing more high-level abstract semantics. Specifically, we present a Self-Supervised Clustering (SSC) loss to preserve more fine-grained semantic clues in exterior representations. It is constituted on the basis of viewing multiple image/text pairs with similar exterior as a category. In addition, a novel harmonious bidirectional triplet ranking (HBTR) loss is proposed, which mitigate the adverse effects brought about by the biased and noisy negative samples. Besides hardest negatives, it also imposes the constraints on the distance between the positive pairs and the centroid of negative pairs. Extensive experimental results on two popular cross-modal retrieval benchmarks demonstrate our proposed MT-HCN can achieve the competitive results compared with the state-of-the-art methods.

Exploiting Hierarchical Activations of Neural Network for Image Retrieval.

Aggregating Hierarchical Binary Activations for Image Retrieval

Exploring Geometric Information in CNN for Image Retrieval.

Learning Feature Embedding with Strong Neural Activations for Fine-Grained Retrieval

Exploiting the Complementary Strengths of Multi-Layer CNN Features for Image Retrieval.

Multi-scale Fusion Transformer Based Weakly Supervised Hashing Learning for Instance Retrieval

Multi-level Image Representation for Large-Scale Image-Based Instance Retrieval.

Multi-Modal Image Fusion Via Deep Laplacian Pyramid Hybrid Network

Multi-task hierarchical convolutional network for visual-semantic cross-modal retrieval

Subspace-based Multi-View Fusion for Instance-Level Image Retrieval

Where to Focus: Query Adaptive Matching for Instance Retrieval Using Convolutional Feature Maps.

Research on image feature extraction and retrieval algorithms based on convolutional neural network

Multi-View Hierarchical Fusion Network for 3D Object Retrieval and Classification

Document Image Retrieval Based on Convolutional Neural Network

Seeing the Big Picture: Deep Embedding with Contextual Evidences

Deep Learning Using Isotroping, Laplacing, Eigenvalues Interpolative Binding, and Convolved Determinants with Normed Mapping for Large-Scale Image Retrieval

Local Deep Descriptors in Bag-of-Words for Image Retrieval.

A multi-level descriptor using ultra-deep feature for image retrieval

Combining Convolutional Neural Network and Markov Random Field for Semantic Image Retrieval

Image Retrieval Algorithm Based on Convolutional Neural Network

Hierarchical Image Retrieval by Multi-Feature Fusion