Abstract:Complex objects are usually with multiple labels, and can be represented by multiple modal representations, e.g., the complex articles contain text and image information as well as multiple annotations. Previous methods assume that the homogeneous multi-modal data are consistent, while in real applications, the raw data are disordered, e.g., the article constitutes with variable number of inconsistent text and image instances. Therefore, Multi-modal Multi-instance Multi-label (M3) learning provides a framework for handling such task and has exhibited excellent performance. However, M3 learning is facing two main challenges: 1) how to effectively utilize label correlation and 2) how to take advantage of multi-modal learning to process unlabeled instances. To solve these problems, we first propose a novel Multi-modal Multi-instance Multi-label Deep Network (M3DN), which considers M3 learning in an end-to-end multi-modal deep network and utilizes consistency principle among different modal bag-level predictions. Based on the M3DN, we learn the latent ground label metric with the optimal transport. Moreover, we introduce the extrinsic unlabeled multi-modal multi-instance data, and propose the M3DNS, which considers the instance-level auto-encoder for single modality and modified bag-level optimal transport to strengthen the consistency among modalities. Thereby M3DNS can better predict label and exploit label correlation simultaneously. Experiments on benchmark datasets and real world WKG Game-Hub dataset validate the effectiveness of the proposed methods.

Complex Object Classification

Semi-Supervised Multi-Modal Multi-Instance Multi-Label Deep Network with Optimal Transport

Collaboration based multi-modal multi-label learning

Multi-Modal Multi-Instance Multi-Label Learning with Graph Convolutional Network

Deep Multimodal Network for Multi-Label Classification.

Multi-View Multi-Instance Multi-Label Learning Based on Collaborative Matrix Factorization.

Multi-typed Objects Multi-view Multi-instance Multi-label Learning.

Multi-label Classification using Deep Multi-order Context-aware Kernel Networks

Multi-Modal Image Annotation with Multi-Instance Multi-Label LDA.

Rethinking Modal-oriented Label Correlations for Multi-modal Multi-label Learning

Multi-Graph-View Learning for Complicated Object Classification.

Weakly-Supervised Multi-view Multi-instance Multi-label Learning

Multi-dimensional Multi-Label Classification: Towards Encompassing Heterogeneous Label Spaces and Multi-Label Annotations

MULTI-LABEL IMAGE RECOGNITION WITH JOINT CLASS-AWARE MAP DISENTANGLING AND LABEL CORRELATION EMBEDDING

Spatial Context-Aware Object-Attentional Network for Multi-Label Image Classification

Multiview Multi-Instance Multilabel Active Learning

Multi-Label Classification with Label Graph Superimposing

Bi-Modal Learning with Channel-Wise Attention for Multi-Label Image Classification

Multimodal Learning for Multi-Label Image Classification.

Object-Based Video Multi-Label Classification with an Improved 3D Convolutional Neural Network

A Unified Modular Framework with Deep Graph Convolutional Networks forMulti-label Image Recognition.