Abstract:Learning common subspace is prevalent way in cross-modal retrieval to solve the problem of data from different modalities having inconsistent distributions and representations that cannot be directly compared. Previous cross-modal retrieval methods focus on projecting the cross-modal data into a common space by learning the correlation between them to bridge the modality gap. However, the rich semantic information in the video and the heterogeneous nature of audio-visual data leads to more serious heterogeneous gaps intuitively, which may lead to the loss of key semantic content of video with single clue by the previous methods when eliminating the modality gap, while the semantics of the categories may undermine the properties of the original features. In this work, we aim to learn effective audio-visual representations to support audio-visual cross-modal retrieval (AVCMR). We propose a novel model that maps audio-visual modalities into two distinct shared latent subspaces: explicit and implicit shared spaces. In particular, the explicit shared space is used to optimize pairwise correlations, where learned representations across modalities capture the commonalities of audio-visual pairs and reduce the modality gap. The implicit shared space is used to preserve the distinctive features between modalities by maintaining the discrimination of audio/video patterns from different semantic categories. Finally, the fusion of the features learned from the two latent subspaces is used for the similarity computation of the AVCMR task. The comprehensive experimental results on two audio-visual datasets demonstrate that our proposed model for using two different latent subspaces for audio-visual cross-modal learning is effective and significantly outperforms the state-of-the-art cross-modal models that learn features from a single subspace.

Hybrid Svm/Hmm Approach For Audio Classification

Hybrid Support Vector Machine and General Model Approach for Audio Classification

Hybrid Independent Component Analysis And Support Vector Machine Approach For Audio Classification

An Approach Based on Reverse Hidden Markov Model for Audio Classification

Hierarchical Support Vector Machines for Audio Classification

HMM-Based Audio Keyword Generation

Audio Sentiment Analysis by Heterogeneous Signal Features Learned from Utterance-Based Parallel Neural Network.

Utterance-Based Audio Sentiment Analysis Learned by a Parallel Combination of CNN and LSTM.

Content-based audio classification and retrieval by support vector machines

A Hybrid Approach To News Video Classification With Multi-Modal Features

A Speech Recognition System Based on a Hybrid HMM/SVM Architecture

Hybrid Independent Component Analysis and Rough Set Approach for Audio Feature Extraction

A Method Based on General Model and Rough Set for Audio Classification

Automatic Music Emotion Classification Using a New Classification Algorithm

New Approach to Classification of Chinese Folk Music Based on Extension of HMM

Integration of Multimodal Features for Video Scene Classification Based on HMM

A Hybrid Approach to Acoustic Scene Classification Based on Universal Acoustic Models.

Audio Analysis Approach based on General Model

SVM+BiHMM: A Hybrid Statistic Model for Metadata Extraction

Learning Explicit and Implicit Latent Common Spaces for Audio-Visual Cross-Modal Retrieval

Automatic evaluation method for vehicle audio warning system using MFCC-polynomial hybrid feature