Counterfactual Attention Alignment for Visible-Infrared Cross-Modality Person Re-Identification

Zongzhe Sun,Feng Zhao

DOI: https://doi.org/10.1016/j.patrec.2023.03.008

IF: 4.757

2023-01-01

Pattern Recognition Letters

Abstract:Visible-infrared person re-identification (VI-ReID) copes with cross-modality matching between the day-time visible and night-time infrared images. Existing methods try to use attention modules to enhance multi-modality feature representations, but ignore measures of attention quality and lack direct and ef-fective supervision of the attention learning process. To solve these problems, we propose a counter-factual attention alignment (CAA) strategy by mining intra-modality attention information with counter-factual causality and aligning the cross-modality attentions. Specifically, a self-weighted part attention module is designed to extract the pairwise attention information in local parts. The counterfactual at-tention alignment strategy obtains the learning results of the attention module through counterfactual intervention, and aligns the attention maps of the two modalities to find better shared cross-modality attention regions. Then the effect of the aligned attention on network prediction is used as a supervision signal to directly guide the attention module to learn more effective attention information. Extensive ex-perimental results demonstrate that the proposed approach outperforms other state-of-the-art methods on two standard benchmarks.(c) 2023 Elsevier B.V. All rights reserved.