Abstract:Cross-media analysis exploits social data with different modalities from multiple sources simultaneously and synergistically to discover knowledge and better understand the world. There are two levels of cross-media social data. One is the element, which is made up of text, images, voice, or any combinations of modalities. Elements from the same data source can have different modalities. The other level of cross-media social data is the new notion of aggregative subject (AS)- a collection of time-series social elements sharing the same semantics (i.e., a collection of tweets, photos, blogs, and news of emergency events). While traditional feature learning methods focus on dealing with single modality data or data fused across multiple modalities, in this study, we systematically analyze the problem of feature learning for cross-media social data at the previously mentioned two levels. The general purpose is to obtain a robust and uniform representation from the social data in time-series and across different modalities. We propose a novel unsupervised method for cross-modality element-level feature learning called cross autoencoder (CAE). CAE can capture the cross-modality correlations in element samples. Furthermore, we extend it to the AS using the convolutional neural network (CNN), namely convolutional cross autoencoder (CCAE). We use CAEs as filters in the CCAE to handle cross-modality elements and the CNN framework to handle the time sequence and reduce the impact of outliers in AS. We finally apply the proposed method to classification tasks to evaluate the quality of the generated representations against several real-world social media datasets. In terms of accuracy, CAE gets 7.33% and 14.31% overall incremental rates on two element-level datasets. CCAE gets 11.2% and 60.5% overall incremental rates on two AS-level datasets. Experimental results show that the proposed CAE and CCAE work well with all tested classifiers and perform better than several other baseline feature learning methods.

Cross-Modality Feature Learning Via Convolutional Autoencoder

CrossMAE: Cross Modality Masked Autoencoders for Region-Aware Audio-Visual Pretraining

Learning Robust Uniform Features for Cross-Media Social Data by Using Cross Autoencoders.

Multiple Kernel Visual-Auditory Representation Learning for Retrieval

Cross-Domain Feature Learning in Multimedia

Cross-Modality Retrieval by Joint Correlation Learning

Cross-modal Semantic Autoencoder with Embedding Consensus

Multimedia Feature Mapping and Correlation Learning for Cross-Modal Retrieval

Correspondence Autoencoders for Cross-Modal Retrieval

A Cross-Modal Image and Text Retrieval Method Based on Efficient Feature Extraction and Interactive Learning CAE

Poster Abstract: Representation Learning from Multimodal Sensor Data with Maximally Correlated Autoencoders

Cross-modal Image-Text Retrieval with Multitask Learning

Deep Memory Network for Cross-Modal Retrieval

Cross-Modal Retrieval Based on Full-Modal Autoencoder with Generative Adversarial Mechanism

Coordinated and specific autoencoder for cross-modal retrieval

Comprehensive Distance-Preserving Autoencoders for Cross-Modal Retrieval

Incomplete Cross-modal Retrieval with Dual-Aligned Variational Autoencoders.

Unsupervised Cross-Modal Retrieval Through Adversarial Learning.

Variational Autoencoder with CCA for Audio-Visual Cross-Modal Retrieval

Multimodal Disentanglement Variational AutoEncoders for Zero-Shot Cross-Modal Retrieval

Multimodal Masked Autoencoders Learn Transferable Representations