Abstract:Cross-modality human behavior analysis has attracted much attention from both academia and industry. In this article, we focus on the cross-modality image-text retrieval problem for human behavior analysis, which can learn a common latent space for cross-modality data and thus benefit the understanding of human behavior with data from different modalities. Existing state-of-the-art cross-modality image-text retrieval models tend to be fine-grained region-word matching approaches, where they begin with measuring similarities for each image region or text word followed by aggregating them to estimate the global image-text similarity. However, it is observed that such fine-grained approaches often encounter the similarity bias problem, because they only consider matched text words for an image region or matched image regions for a text word for similarity calculation, but they totally ignore unmatched words/regions, which might still be salient enough to affect the global image-text similarity. In this article, we propose an Adaptive Confidence Matching Network (ACMNet), which is also a fine-grained matching approach, to effectively deal with such a similarity bias. Apart from calculating the local similarity for each region(/word) with its matched words(/regions), ACMNet also introduces a confidence score for the local similarity by leveraging the global text(/image) information, which is expected to help measure the semantic relatedness of the region(/word) to the whole text(/image). Moreover, ACMNet also incorporates the confidence scores together with the local similarities in estimating the global image-text similarity. To verify the effectiveness of ACMNet, we conduct extensive experiments and make comparisons with state-of-the-art methods on two benchmark datasets, i.e., Flickr30k and MS COCO. Experimental results show that the proposed ACMNet can outperform the state-of-the-art methods by a clear margin, which well demonstrates the effectiveness of the proposed ACMNet in human behavior analysis and the reasonableness of tackling the mentioned similarity bias issue.

Comparison of Human Face Matching Behavior and Computational Image Similarity Measure.

ACMNet

3D Face Similarity Measurement Based on Adaptive Neighborhood Description

Design and Similarity Evaluation on Humanoid Facial Expression

Evaluating Representational Similarity Measures from the Lens of Functional Correspondence

Which Face is More Attractive?

Face identification using reference-based features with message passing model

Face Recognition under Occlusions and Variant Expressions with Partial Similarity

HOS Image Similarity Measure for Human Face Recognition

Novel similarity measures for face representation based on local binary pattern

A Face Recognition Workflow Based Upon Similarity Measurement.

A Feature-Based Structural Measure: an Image Similarity Measure for Face Recognition

Face matching as a majority: Getting the best from a crowd

Comparison System of Face Identification Algorithms under Application-Specific

A Comparison of Methods of Facial Expression Recognition

Face recognition based on non-corresponding region matching

3D Facial Similarity Comparison in Shape Space

The Analysis of Similarity Measure Function in Image Matching Algorithms

The Comparative Research and Implementation of based on Gabor Features Face Recognition Algorithms

Comparing Human and Machine Bias in Face Recognition

3D Facial Similarity Measure Based on Geodesic Network and Curvatures