Multiview Detection with Shadow Transformer (and View-Coherent Data Augmentation)

Yunzhong Hou,Liang Zheng

DOI: https://doi.org/10.48550/arXiv.2108.05888

2021-08-13

Abstract:Multiview detection incorporates multiple camera views to deal with occlusions, and its central problem is multiview aggregation. Given feature map projections from multiple views onto a common ground plane, the state-of-the-art method addresses this problem via convolution, which applies the same calculation regardless of object locations. However, such translation-invariant behaviors might not be the best choice, as object features undergo various projection distortions according to their positions and cameras. In this paper, we propose a novel multiview detector, MVDeTr, that adopts a newly introduced shadow transformer to aggregate multiview information. Unlike convolutions, shadow transformer attends differently at different positions and cameras to deal with various shadow-like distortions. We propose an effective training scheme that includes a new view-coherent data augmentation method, which applies random augmentations while maintaining multiview consistency. On two multiview detection benchmarks, we report new state-of-the-art accuracy with the proposed system. Code is available at <a class="link-external link-https" href="https://github.com/hou-yz/MVDeTr" rel="external noopener nofollow">this https URL</a>.

Computer Vision and Pattern Recognition

What problem does this paper attempt to address?

This paper attempts to solve the multi - view aggregation problem in multi - view detection. Specifically, existing methods usually use convolution to process feature maps projected from multiple views onto the common ground plane, but this method exhibits translational invariance under different positions and cameras, that is, the same calculation method is applied to all positions. However, since object features will be distorted in various ways according to their positions and cameras during the projection process, this translation - invariant behavior may not be the best choice. Therefore, the paper proposes a new multi - view detector MVDeTr, which uses the newly introduced Shadow Transformer to aggregate multi - view information. The Shadow Transformer can allocate different attention according to different positions and cameras to deal with various shadow - like distortions. In addition, the paper also proposes an effective training scheme, including a new view - consistency data augmentation method, which applies random augmentation while maintaining multi - view consistency. Through these improvements, the paper reports new state - of - the - art accuracies on two multi - view detection benchmarks.