Improving Multi-label Learning with Missing Labels by Structured Semantic Correlations

Hao Yang,Joey Tianyi Zhou,Jianfei Cai

DOI: https://doi.org/10.48550/arXiv.1608.01441

2016-08-04

Abstract:Multi-label learning has attracted significant interests in computer vision recently, finding applications in many vision tasks such as multiple object recognition and automatic image annotation. Associating multiple labels to a complex image is very difficult, not only due to the intricacy of describing the image, but also because of the incompleteness nature of the observed labels. Existing works on the problem either ignore the label-label and instance-instance correlations or just assume these correlations are linear and unstructured. Considering that semantic correlations between images are actually structured, in this paper we propose to incorporate structured semantic correlations to solve the missing label problem of multi-label learning. Specifically, we project images to the semantic space with an effective semantic descriptor. A semantic graph is then constructed on these images to capture the structured correlations between them. We utilize the semantic graph Laplacian as a smooth term in the multi-label learning formulation to incorporate the structured semantic correlations. Experimental results demonstrate the effectiveness of the proposed semantic descriptor and the usefulness of incorporating the structured semantic correlations. We achieve better results than state-of-the-art multi-label learning methods on four benchmark datasets.

Computer Vision and Pattern Recognition

What problem does this paper attempt to address?

The problem that this paper attempts to solve is dealing with the situation of missing labels in multi - label learning. Specifically, the paper points out that in multi - label learning tasks, it is very difficult to assign multiple labels to complex images, not only because of the complexity of describing the images themselves, but also because the observed labels are often incomplete. Existing methods either ignore the correlations between labels and the correlations between instances, or assume that these correlations are linear and unstructured. However, in practical applications, especially in image recognition, the correlations between labels and the correlations between instances are actually structured. Therefore, the paper proposes a method to solve the problem of missing labels in multi - label learning by combining structured semantic correlations.

The main contributions of the paper lie in proposing a new semantic representation method and a method of incorporating structured semantic correlations into multi - label learning. By constructing a semantic graph and using the semantic graph Laplacian as a smoothing term, the method in the paper can more effectively capture the structured correlations between instances, thereby improving the effect of multi - label learning. Experimental results show that this method significantly outperforms existing multi - label learning methods on four benchmark datasets, especially when the training label observation rate is low (for example, only 10% of the given training labels are observed).