Abstract:Learning common subspace is prevalent way in cross-modal retrieval to solve the problem of data from different modalities having inconsistent distributions and representations that cannot be directly compared. Previous cross-modal retrieval methods focus on projecting the cross-modal data into a common space by learning the correlation between them to bridge the modality gap. However, the rich semantic information in the video and the heterogeneous nature of audio-visual data leads to more serious heterogeneous gaps intuitively, which may lead to the loss of key semantic content of video with single clue by the previous methods when eliminating the modality gap, while the semantics of the categories may undermine the properties of the original features. In this work, we aim to learn effective audio-visual representations to support audio-visual cross-modal retrieval (AVCMR). We propose a novel model that maps audio-visual modalities into two distinct shared latent subspaces: explicit and implicit shared spaces. In particular, the explicit shared space is used to optimize pairwise correlations, where learned representations across modalities capture the commonalities of audio-visual pairs and reduce the modality gap. The implicit shared space is used to preserve the distinctive features between modalities by maintaining the discrimination of audio/video patterns from different semantic categories. Finally, the fusion of the features learned from the two latent subspaces is used for the similarity computation of the AVCMR task. The comprehensive experimental results on two audio-visual datasets demonstrate that our proposed model for using two different latent subspaces for audio-visual cross-modal learning is effective and significantly outperforms the state-of-the-art cross-modal models that learn features from a single subspace.

Predicting Content Similarity Via Multimodal Modeling for Video-In-Video Advertising.

An Unsupervised Video Summarization Method Based on Multimodal Representation.

Videoader: a video advertising system based on intelligent analysis of visual content

Multimodal Content Analysis for Effective Advertisements on YouTube

Contrastive Intra- and Inter-Modality Generation for Enhancing Incomplete Multimedia Recommendation

Multi-modal Representation Learning for Video Advertisement Content Structuring

A Multimodal Framework for Video Ads Understanding

Smart Advertising in Videos Based on Comprehensive Content Analytics

Multi-layer multi-view topic model for classifying advertising video.

Predicting Online Video Advertising Effects with Multimodal Deep Learning

MM-AU:Towards Multimodal Understanding of Advertisement Videos

Look, Read and Feel: Benchmarking Ads Understanding with Multimodal Multitask Learning

Learning Explicit and Implicit Latent Common Spaces for Audio-Visual Cross-Modal Retrieval

Video Ads Content Structuring by Combining Scene Confidence Prediction and Tagging

ContextIQ: A Multimodal Expert-Based Video Retrieval System for Contextual Advertising

Implicit semantic-based personalized micro-videos recommendation

Multimodal Adversarially Learned Inference with Factorized Discriminators

Automatic Generation of Interactive Nonlinear Video for Online Apparel Shopping Navigation

Multimodal Marketing Intent Analysis for Effective Targeted Advertising

Linking Video ADS with Product or Service Information by Web Search

Mining Adjacent Markets from a Large-Scale Ads Video Collection for Image Advertising