Abstract:6DoF pose estimation has received much attention in recent years. A key challenge is the difficulty of estimating object pose when the target texture is weak. In this work, we present the cross-modal Transformer (CMT-6D), a Transformer-based network suitable for highly accurate workpiece-level object 6D pose estimation from a single RGBD image. Our main insight is to make the surface texture information of RGB images with the geometric feature information of point clouds complement each other through a cross-modal Transformer, enabling accurate estimation of the pose of weakly textured targets. Specifically, the whole framework consists of two parallel Transformer branches, named Point Transformer and Image Transformer. Both parallel transformer networks use a pyramid structured encoder and a multi-layer perceptron structured decoder to extract geometric features of point clouds and texture features of RGB images, respectively. Then, a cross-modal key query strategy is proposed for information exchange between parallel channels. In addition, at the output representation stage, we design a simple and effective 3D keypoint selection algorithm to solve the problem that keypoints are likely to appear in the non-significant region. Finally, to improve the accuracy of attitude estimation and meet real-time requirements, a lightweight pose iterative network based on target feature regression is proposed to correct the initial attitude estimation error. Extensive experiments demonstrate the effectiveness and superiority of our method on LineMOD, Occlusion LineMOD, T-Less, and YCB-Video datasets. We demonstrate that our method can improve the 6D pose estimation performance by comparing with the state-of-the-art. Ablation research and visualization validate the design of CMT-6D.

A Transformer-based multi-modal fusion network for 6D pose estimation

MoreFusion: Multi-object Reasoning for 6D Pose Estimation from Volumetric Fusion

Recurrent Volume-based 3D Feature Fusion for Real-time Multi-view Object Pose Estimation

An Iterative Attention Fusion Network for 6D Object Pose Estimation

3D Point-to-Keypoint Voting Network for 6D Pose Estimation

Recurrent Volume-Based 3-D Feature Fusion for Real-Time Multiview Object Pose Estimation.

6D Object Pose Estimation Based on Cross-Modality Feature Fusion

A modal fusion network with dual attention mechanism for 6D pose estimation

An Efficient Color and Geometric Feature Fusion Module for 6D Object Pose Estiamtion

Homologous multimodal fusion network with geometric constraint keypoints selection for 6D pose estimation

Semi-Decoupled 6D Pose Estimation Via Multi-Modal Feature Fusion

CMT-6D: a lightweight iterative 6DoF pose estimation network based on cross-modal Transformer

MixedFusion: 6D Object Pose Estimation from Decoupled RGB-Depth Features.

ACF-Net: Attention Context Fusion Network for 6D Pose Estimation

FEIF: Feature Excitation and Interactive Fusion for 6D Object Pose Estimation.

A Lightweight Color and Geometry Feature Extraction and Fusion Module for End-to-end 6D Pose Estimation

Robust Classification and 6D Pose Estimation by Sensor Dual Fusion of Image and Point Cloud Data

6-D Object Pose Estimation Using Multiscale Point Cloud Transformer.

A Novel Depth and Color Feature Fusion Framework for 6D Object Pose Estimation.

Cross-modal Attention and Geometric Contextual Aggregation Network for 6dof Object Pose Estimation

DenseFusion: 6D Object Pose Estimation by Iterative Dense Fusion