Multi-modal constraint propagation via compatible conditional distribution reconstruction

Yaoyi Li,Hongtao Lu
DOI: https://doi.org/10.1016/j.neucom.2020.09.067
IF: 6
2021-01-01
Neurocomputing
Abstract:In many applications, the pairwise constraint is a kind of weaker supervisory information which can be collected more efficiently than the label information. The constraint propagation is a constrained clustering approach, which has been proved to be a success of exploiting such side-information. Recent years have witnessed many methods of multi-modal constraint propagation. However, the problem of reasonably fusing different modalities under the framework of constraint propagation remains unaddressed. In this paper, we first identify a necessary and sufficient condition for compatible conditional distributions under a specific assumption and address the problem of Compatible Conditional Distributions Reconstruction (CCDR). With the help of CCDR, we propose a multi-modal constraint propagation method dubbed Instance Level Multi-Modal Constraint Propagation (ILMCP). ILMCP fuses the affinity of different modalities at the data instance level instead of the modality level and constructs a unified affinity matrix. Extensive experiments on two publicly available multi-modal datasets show the superior performance of the proposed method.
What problem does this paper attempt to address?