Distilling Localization for Self-Supervised Representation Learning

Nanxuan Zhao,Zhirong Wu,Rynson W.H. Lau,Stephen Lin

DOI: https://doi.org/10.48550/arXiv.2004.06638

2021-01-19

Abstract:Recent progress in contrastive learning has revolutionized unsupervised representation learning. Concretely, multiple views (augmentations) from the same image are encouraged to map to the similar embeddings, while views from different images are pulled apart. In this paper, through visualizing and diagnosing classification errors, we observe that current contrastive models are ineffective at localizing the foreground object, limiting their ability to extract discriminative high-level features. This is due to the fact that view generation process considers pixels in an image uniformly. To address this problem, we propose a data-driven approach for learning invariance to backgrounds. It first estimates foreground saliency in images and then creates augmentations by copy-and-pasting the foreground onto a variety of backgrounds. The learning still follows the instance discrimination pretext task, so that the representation is trained to disregard background content and focus on the foreground. We study a variety of saliency estimation methods, and find that most methods lead to improvements for contrastive learning. With this approach (DiLo), significant performance is achieved for self-supervised learning on ImageNet classification, and also for object detection on PASCAL VOC and MSCOCO.

Computer Vision and Pattern Recognition

What problem does this paper attempt to address?

The problem that this paper attempts to solve is the insufficient ability of current models in localizing foreground objects in self - supervised representation learning. Specifically, through visualization and diagnosis of classification errors, the author found that existing contrastive learning models cannot effectively localize foreground objects when processing images, resulting in the features they extract lacking in distinctiveness. This is because the process of generating views uniformly considers every pixel in the image without paying special attention to the foreground objects, and the background information is often similar among different classes, which makes the model easily interfered by the background information.

To solve this problem, the paper proposes a data - driven method to learn background invariance. That is, first estimate the foreground saliency in the image, and then paste the foreground objects onto different backgrounds by copy - and - paste to create augmented views. This method still follows the instance discrimination pre - training task, so that the trained representation can ignore the background content and focus on the foreground objects. In this way, the DiLo method proposed in the paper has achieved significant performance improvements on ImageNet classification and PASCAL VOC, MSCOCO object detection tasks.