Abstract:Incorporating multiple camera views for detection alleviates the impact of occlusions in crowded scenes. In a multiview system, we need to answer two important questions when dealing with ambiguities that arise from occlusions. First, how should we aggregate cues from the multiple views? Second, how should we aggregate unreliable 2D and 3D spatial information that has been tainted by occlusions? To address these questions, we propose a novel multiview detection system, MVDet. For multiview aggregation, existing methods combine anchor box features from the image plane, which potentially limits performance due to inaccurate anchor box shapes and sizes. In contrast, we take an anchor-free approach to aggregate multiview information by projecting feature maps onto the ground plane (bird's eye view). To resolve any remaining spatial ambiguity, we apply large kernel convolutions on the ground plane feature map and infer locations from detection peaks. Our entire model is end-to-end learnable and achieves 88.2% MODA on the standard Wildtrack dataset, outperforming the state-of-the-art by 14.1%. We also provide detailed analysis of MVDet on a newly introduced synthetic dataset, MultiviewX, which allows us to control the level of occlusion. Code and MultiviewX dataset are available at <a class="link-external link-https" href="https://github.com/hou-yz/MVDet" rel="external noopener nofollow">this https URL</a>.

What problem does this paper attempt to address?

This paper attempts to solve the occlusion problem in multi - view pedestrian detection. Specifically, it mainly focuses on two key issues: 1. **How to aggregate information from multiple views?** - The paper points out that in multi - view detection systems, existing methods usually use predefined anchor boxes to represent each position on the ground, which may limit performance because these predefined anchor boxes may be inaccurate. To solve this problem, the paper proposes a new multi - view detector (MVDet), which directly samples feature vectors from pixels corresponding to multiple views through perspective transformation of feature maps, thereby achieving an anchor - free representation. 2. **How to aggregate information of spatially adjacent positions?** - For spatial aggregation, traditional multi - view detection methods usually need to be designed and operated outside the convolutional neural network (CNN), such as conditional random fields (CRF) or mean - field inference. MVDet adopts a fully convolutional method, by applying large convolution kernels on the feature map after multi - view aggregation, thus jointly considering the information of spatially adjacent positions. By solving these two problems, MVDet achieves 88.2% MODA (Multi - Object Detection Accuracy) on the Wildtrack dataset, which is 14.1% higher than the previous best method. In addition, the paper also introduces a new synthetic dataset MultiviewX to control the degree of occlusion and conducts a detailed analysis on this dataset. ### Main contributions 1. **Anchor - free multi - view aggregation**: - Through perspective transformation of feature maps, MVDet can directly sample feature vectors from pixels corresponding to multiple views, avoiding the inaccuracy brought by predefined anchor boxes. 2. **Fully convolutional spatial aggregation**: - By applying large convolution kernels on the feature map after multi - view aggregation, MVDet can jointly consider the information of spatially adjacent positions within the network without additional design and operation. 3. **High performance**: - MVDet achieves significant performance improvements on both the Wildtrack and MultiviewX datasets, especially in severely occluded scenes. ### Formulas and methods - **Perspective transformation**: - The paper uses the perspective transformation matrix \( P_\theta \) to map 3D world coordinates \((x, y, z)\) to 2D image coordinates \((u, v)\): \[ s \begin{pmatrix} u \\ v \\ 1 \end{pmatrix} = P_\theta \begin{pmatrix} x \\ y \\ z \\ 1 \end{pmatrix} = A [R | t] \begin{pmatrix} x \\ y \\ z \\ 1 \end{pmatrix} = \begin{pmatrix} \theta_{11} & \theta_{12} & \theta_{13} & \theta_{14} \\ \theta_{21} & \theta_{22} & \theta_{23} & \theta_{24} \\ \theta_{31} & \theta_{32} & \theta_{33} & \theta_{34} \end{pmatrix} \begin{pmatrix} x \\ y \\ z \\ 1 \end{pmatrix} \] - When \( z = 0 \), it can be simplified to: \[ s \begin{pmatrix} u \\ v \\ 1 \end{pmatrix} = P_{\theta,0} \begin{pmatrix} x \\ y \\ 1 \end{pmatrix} = \beg

Multiview Detection with Feature Perspective Transformation

Multi-View Domain Adaptive Object Detection on Camera Networks.

Multilevel Spatial-Temporal Feature Aggregation for Video Object Detection

Multiview Detection with Shadow Transformer (and View-Coherent Data Augmentation)

MVM3Det: A Novel Method for Multi-view Monocular 3D Detection

Query-Based Multiview Detection for Multiple Visual Sensor Networks

Learning to Learn Multiview Detection by Camera-Aware Attention

3M3D: Multi-view, Multi-path, Multi-representation for 3D Object Detection

MVMM: Multiview Multimodal 3-D Object Detection for Autonomous Driving

Voxelized 3D Feature Aggregation for Multiview Detection

Viewpoint Equivariance for Multi-View 3D Object Detection

Multi-view Aggregation for Real-Time Accurate Object Detection of a Moving Camera

DVPE: Divided View Position Embedding for Multi-View 3D Object Detection

Towards Generalizable Multi-Camera 3D Object Detection via Perspective Debiasing

Multi-View People Detection in Large Scenes via Supervised View-Wise Contribution Weighting

A Multi-view 3D Vehicle Detection Method Based On Novel 3D Proposal Generation Method

Scaling Multi-Camera 3D Object Detection through Weak-to-Strong Eliciting

Multiview Detection with Cardboard Human Modeling

End-to-End Multi-View Fusion for 3D Object Detection in LiDAR Point Clouds

Unsupervised Multi-view Pedestrian Detection

Novel Framework for Multi-view Object Detection through Combining Multiple Classifiers