Spatial-Aware Token for Weakly Supervised Object Localization

Pingyu Wu,Wei Zhai,Yang Cao,Jiebo Luo,Zheng-Jun Zha

2023-08-09

Abstract:Weakly supervised object localization (WSOL) is a challenging task aiming to localize objects with only image-level supervision. Recent works apply visual transformer to WSOL and achieve significant success by exploiting the long-range feature dependency in self-attention mechanism. However, existing transformer-based methods synthesize the classification feature maps as the localization map, which leads to optimization conflicts between classification and localization tasks. To address this problem, we propose to learn a task-specific spatial-aware token (SAT) to condition localization in a weakly supervised manner. Specifically, a spatial token is first introduced in the input space to aggregate representations for localization task. Then a spatial aware attention module is constructed, which allows spatial token to generate foreground probabilities of different patches by querying and to extract localization knowledge from the classification task. Besides, for the problem of sparse and unbalanced pixel-level supervision obtained from the image-level label, two spatial constraints, including batch area loss and normalization loss, are designed to compensate and enhance this supervision. Experiments show that the proposed SAT achieves state-of-the-art performance on both CUB-200 and ImageNet, with 98.45% and 73.13% GT-known Loc, respectively. Even under the extreme setting of using only 1 image per class from ImageNet for training, SAT already exceeds the SOTA method by 2.1% GT-known Loc. Code and models are available at <a class="link-external link-https" href="https://github.com/wpy1999/SAT" rel="external noopener nofollow">this https URL</a>.

Computer Vision and Pattern Recognition

What problem does this paper attempt to address?

The paper aims to address the optimization conflict problem in the task of Weakly Supervised Object Localization (WSOL). Specifically, existing methods based on vision transformers have achieved significant success in capturing long-range feature dependencies using self-attention mechanisms. However, these methods synthesize classification feature maps as localization maps, leading to optimization conflicts between classification and localization tasks. To solve this problem, the authors propose a task-specific Spatial-Aware Token (SAT) to conditionally perform weakly supervised localization. By introducing spatial tokens to aggregate representations and constructing a spatial-aware attention module, the spatial tokens can generate foreground probabilities for different patches and extract localization knowledge from the classification task.

Moreover, to address the issue of sparse and imbalanced pixel-level supervision obtained from image-level labels, the paper designs two spatial constraints, namely batch area loss and normalization loss, to compensate and enhance this supervision. Experimental results show that the proposed SAT method achieves state-of-the-art performance on the CUB-200 and ImageNet datasets, reaching 98.45% and 73.13% on the GT-known Loc metric, respectively. Even under extreme conditions (using only 1 training image per class), SAT outperforms the existing best method by 2.1% in GT-known Loc. This approach not only improves localization accuracy but also demonstrates efficient data usage and adjustment efficiency.