Abstract:Point cloud-based 3D object tracking is an important task in autonomous driving. Though great advances regarding Siamese-based 3D tracking have been made recently, it remains challenging to learn the correlation between the template and search branches effectively with the sparse LIDAR point cloud data. Instead of performing correlation of the two branches at just one point in the network, in this paper, we present a multi-correlation Siamese Transformer network that has multiple stages and carries out feature correlation at the end of each stage based on sparse pillars. More specifically, in each stage, self-attention is first applied to each branch separately to capture the non-local context information. Then, cross-attention is used to inject the template information into the search area. This strategy allows the feature learning of the search area to be aware of the template while keeping the individual characteristics of the template intact. To enable the network to easily preserve the information learned at different stages and ease the optimization, for the search area, we densely connect the initial input sparse pillars and the output of each stage to all subsequent stages and the target localization network, which converts pillars to bird's eye view (BEV) feature maps and predicts the state of the target with a small densely connected convolution network. Deep supervision is added to each stage to further boost the performance as well. The proposed algorithm is evaluated on the popular KITTI, nuScenes, and Waymo datasets, and the experimental results show that our method achieves promising performance compared with the state-of-the-art. Ablation study that shows the effectiveness of each component is provided as well. Code is available at <a class="link-external link-https" href="https://github.com/liangp/MCSTN-3DSOT" rel="external noopener nofollow">this https URL</a>.

What problem does this paper attempt to address?

The paper is primarily dedicated to addressing the challenging issues in 3D Single Object Tracking (3D SOT), particularly in the context of autonomous driving, where effective tracking is performed using sparse point cloud data generated by LIDAR. Specifically, the paper points out that although methods based on the Siamese architecture have made significant progress in 3D object tracking, effectively learning the association between the template branch and the search branch on sparse LIDAR point cloud data remains a challenge. Most existing methods only fuse the information of these two branches at a single network node, which either sacrifices the semantic information integrity of the template branch or lacks sufficient template information awareness during the feature extraction process of the search area. To address the above issues, the paper proposes a multi-association Siamese Transformer network with multiple stages, performing feature association based on sparse pillars at the end of each stage. Specifically, the network first independently processes the two branches using self-attention to capture non-local contextual information, and then uses cross-attention to inject template information into the search area. Additionally, to facilitate information transfer between different stages of the network and simplify the optimization process, the paper adopts a dense connection strategy, where the initial input pillar features and the output of each stage are densely connected to all subsequent stages and the final target localization network input. In this way, the paper addresses the following core issues: 1. **Maintaining the integrity of the template branch**: Ensuring that the semantic information of the template branch is preserved throughout the network. 2. **Enhancing the search area's awareness of template information**: Ensuring that the feature extraction process of the search area fully considers the template information. 3. **Improving the feature fusion mechanism**: Enhancing the information interaction between the template and search branches by performing feature fusion at multiple network nodes. 4. **Simplifying the optimization process**: Using a dense connection strategy to promote the flow of features between different network layers, thereby simplifying the network training process. Experimental results show that the proposed algorithm outperforms the current state-of-the-art methods on popular datasets such as KITTI, nuScenes, and Waymo, demonstrating its effectiveness.

Multi-Correlation Siamese Transformer Network with Dense Connection for 3D Single Object Tracking

3D Siamese Transformer Network for Single Object Tracking on Point Clouds

Exploit Spatiotemporal Contextual Information for 3D Single Object Tracking Via Memory Networks

Multi-modal 3D Human Tracking for Robots in Complex Environment with Siamese Point-Video Transformer

Beyond Traditional Driving Scenes: A Robotic-Centric Paradigm for 2D+3D Human Tracking Using Siamese Transformer Network

Correlation Pyramid Network for 3D Single Object Tracking

OST: Efficient One-stream Network for 3D Single Object Tracking in Point Clouds

Multitarget Tracking Using Siamese Neural Networks

Synchronize Feature Extracting and Matching: A Single Branch Framework for 3D Object Tracking

3D Object Tracking with Transformer

The Multi-task Fully Convolutional Siamese Network with Correlation Filter Layer for Real-Time Visual Tracking

Multi-Level Structure-Enhanced Network for 3D Single Object Tracking in Sparse Point Clouds

Correlation-Embedded Transformer Tracking: A Single-Branch Framework

3D-SiamRPN: An End-to-End Learning Method for Real-Time 3D Single Object Tracking Using Raw Point Cloud

Point Cloud Registration-Driven Robust Feature Matching for 3-D Siamese Object Tracking

CTT: CNN Meets Transformer for Tracking

Hierarchical correlation siamese network for real-time object tracking

Exploiting More Information in Sparse Point Cloud for 3D Single Object Tracking

TICNet: A Target-Insight Correlation Network for Object Tracking

Temporal-Aware Siamese Tracker: Integrate Temporal Context for 3D Object Tracking