Abstract:Cross-modality human behavior analysis has attracted much attention from both academia and industry. In this article, we focus on the cross-modality image-text retrieval problem for human behavior analysis, which can learn a common latent space for cross-modality data and thus benefit the understanding of human behavior with data from different modalities. Existing state-of-the-art cross-modality image-text retrieval models tend to be fine-grained region-word matching approaches, where they begin with measuring similarities for each image region or text word followed by aggregating them to estimate the global image-text similarity. However, it is observed that such fine-grained approaches often encounter the similarity bias problem, because they only consider matched text words for an image region or matched image regions for a text word for similarity calculation, but they totally ignore unmatched words/regions, which might still be salient enough to affect the global image-text similarity. In this article, we propose an Adaptive Confidence Matching Network (ACMNet), which is also a fine-grained matching approach, to effectively deal with such a similarity bias. Apart from calculating the local similarity for each region(/word) with its matched words(/regions), ACMNet also introduces a confidence score for the local similarity by leveraging the global text(/image) information, which is expected to help measure the semantic relatedness of the region(/word) to the whole text(/image). Moreover, ACMNet also incorporates the confidence scores together with the local similarities in estimating the global image-text similarity. To verify the effectiveness of ACMNet, we conduct extensive experiments and make comparisons with state-of-the-art methods on two benchmark datasets, i.e., Flickr30k and MS COCO. Experimental results show that the proposed ACMNet can outperform the state-of-the-art methods by a clear margin, which well demonstrates the effectiveness of the proposed ACMNet in human behavior analysis and the reasonableness of tackling the mentioned similarity bias issue.

ART2 neural network with bidirectional matching mechanism

An Enhanced ART2 Neural Network for Clustering Analysis

A novel pattern recognition algorithm: Combining ART network with SVM to reconstruct a multi-class classifier

A new ART neural networks for remote sensing image classification

ACMNet

The Research on a Fractionizing and Fitting ART2 Neural Network with Supervise

An improved ART 2-A model for mixed numeric and categorical data

An Algorithm of Expressway Automatic Incident Detection Based on ART2 Neural Networks

A Novel Neural Network for Remote Sensing Image Matching

Adjoint Rigid Transform Network: Task-conditioned Alignment of 3D Shapes

Reference-Aware Adaptive Network for Image-Text Matching

Learning Two-Branch Neural Networks for Image-Text Matching Tasks.

A Detector-oblivious Multi-arm Network for Keypoint Matching

Pattern recognition using asymmetric attractor neural networks

Research of field theory based adaptive resonance neural network

A Survey of Adaptive Resonance Theory Neural Network Models for Engineering Applications

AN ADAPTIVE FAULT-TOLERANT NEURAL NETWORK CLASSIFIER

CSR-Net: Learning Adaptive Context Structure Representation for Robust Feature Correspondence

An unsupervised approach based on ART network for coreference resolution of Chinese

Artificial Retinal Neural Network For Visual Pattern Recognition

The Performance of Two CNN Methods in Artworks Aesthetic Feature Recognition