Abstract:3D human tracking plays a crucial role in the automation intelligence system. Current approaches focus on achieving higher performance on traditional driving datasets like KITTI, which overlook the jitteriness of the platform and the complexity of the environments. Once the scenarios are migrated to jolting robot platforms, they all degenerate severely with only a 20-60% success rate, which greatly restricts the high-level application of autonomous systems. In this work, beyond traditional flat scenes, we introduce Multi-modal Human Tracking Paradigm (MHTrack), a unified multimodal transformer-based model that can effectively track the target person frame-by-frame in point and video sequences. Specifically, we design a speed-inertia module-assisted stabilization mechanism along with an alternate training strategy to better migrate the tracking algorithm to the robot platform. To capture more target-aware information, we combine the geometric and appearance features of point clouds and video frames together based on a hierarchical Siamese Transformer Network. Additionally, considering the prior characteristics of the human category, we design a lateral cross-attention pyramid head for deeper feature aggregation and final 3D BBox generation. Extensive experiments confirm that MHTrack significantly outperforms the previous state-of-the-arts on both open-source datasets and large-scale robotic datasets. Further analysis verifies each component’s effectiveness and shows the robotic-centric paradigm’s promising potential when deployed into dynamic robotic systems.

PHD: A Deep Learning Based Human Detection Framework for Panoramic Videos

Video Based Head Detection and Tracking Surveillance System

Real-Time Drone Detection Using Deep Learning Approach.

Beyond Traditional Driving Scenes: A Robotic-Centric Paradigm for 2D+3D Human Tracking Using Siamese Transformer Network

Large Area 3D Human Pose Detection Via Stereo Reconstruction in Panoramic Cameras

Detecting and tracking people in real time with RGB-D camera

A Panoramic Video Face Detection System Design and Implement

People Tracking in Panoramic Video for Guiding Robots

Robust 3-D Human Detection in Complex Environments With a Depth Camera.

Human Detection in Video over Large Viewpoint Changes

A Multi-view 3D Vehicle Detection Method Based On Novel 3D Proposal Generation Method

3D Object Detection Algorithm for Panoramic Images with Multi-Scale Convolutional Neural Network.

Fast Human Detection in RGB-D Images Based on Color-Depth Joint Feature Learning.

Motion Detection for Human Bodies Basing Adaptive Background Subtraction by Using an Omnidirectional Camera

GigaHumanDet: Exploring Full-Body Detection on Gigapixel-Level Images

Detection and tracking of human track and field motion targets based on deep learning

Using Panoramic Videos for Multi-person Localization and Tracking in a 3D Panoramic Coordinate

Robust Real-Time Human Perception with Depth Camera.

Incorporation of Panoramic View in Fall Detection Using Omnidirectional Camera

A Unified Framework for Human-centric Point Cloud Video Understanding

Real-time human detection and tracking in complex environments using single RGBD camera