Abstract:3D human tracking plays a crucial role in the automation intelligence system. Current approaches focus on achieving higher performance on traditional driving datasets like KITTI, which overlook the jitteriness of the platform and the complexity of the environments. Once the scenarios are migrated to jolting robot platforms, they all degenerate severely with only a 20-60% success rate, which greatly restricts the high-level application of autonomous systems. In this work, beyond traditional flat scenes, we introduce Multi-modal Human Tracking Paradigm (MHTrack), a unified multimodal transformer-based model that can effectively track the target person frame-by-frame in point and video sequences. Specifically, we design a speed-inertia module-assisted stabilization mechanism along with an alternate training strategy to better migrate the tracking algorithm to the robot platform. To capture more target-aware information, we combine the geometric and appearance features of point clouds and video frames together based on a hierarchical Siamese Transformer Network. Additionally, considering the prior characteristics of the human category, we design a lateral cross-attention pyramid head for deeper feature aggregation and final 3D BBox generation. Extensive experiments confirm that MHTrack significantly outperforms the previous state-of-the-arts on both open-source datasets and large-scale robotic datasets. Further analysis verifies each component’s effectiveness and shows the robotic-centric paradigm’s promising potential when deployed into dynamic robotic systems.

Adaptive Visual Trajectory Tracking of Nonholonomic Mobile Robots Based on Trifocal Tensor

Adaptive Model Prediction Control-Based Multi-Terrain Trajectory Tracking Framework for Mobile Spherical Robots

Vision-based Adaptive Tracking Control of Uncertain Nonholonomic Mobile Robots

Adaptive Trajectory Tracking of Nonholonomic Mobile Robots Using Vision-Based Position and Velocity Estimation

Beyond Traditional Driving Scenes: A Robotic-Centric Paradigm for 2D+3D Human Tracking Using Siamese Transformer Network

Accurate and Real-Time 3-D Tracking for the Following Robots by Fusing Vision and Ultrasonar Information

An Enhanced Trajectory Optimization Approach for Equirectangular Camera-based Visual SLAM

Visual Trajectory Tracking of Wheeled Mobile Robots with Uncalibrated Camera Extrinsic Parameters.

Vision-based Trajectory Tracking of Mobile Robot

Trifocal tensor-based 6-DOF visual servoing

Trajectory Planning for Visual Servoing with Some Constraints

Visual Tracking Control of General Robotic Systems

Estimation of Extrinsic Parameters with Trifocal Tensor for Intelligent Vehicle-Mounted Cameras

Asymptotic Moving Object Tracking with Trajectory Tracking Extension: A Homography‐based Approach

Visual Tracking and Depth Estimation of Mobile Robots Without Desired Velocity Information.

Adaptive Trajectory Tracking of Wheeled Mobile Robots Based on a Fish-eye Camera

Adaptive homography-based visual servo tracking.

Unified Tracking and Regulation Visual Servoing of Wheeled Mobile Robots with Euclidean Reconstruction.

Adaptive Homography-Based Visual Servo Tracking for a Fixed Camera Configuration with a Camera-in-hand Extension

Smooth-optimal Adaptive Trajectory Tracking Using an Uncalibrated Fish-eye Camera

Mobile Target Tracking Based on Hybrid Open-Loop Monocular Vision Motion Control Strategy