Abstract:Transparent objects are commonly found in real life and industrial production. Unlike opaque objects, transparent objects are not easily identifiable in RGB images and often require depth information to determine their position in the image. However, due to the influence of other environmental factors such as reflection and refraction, the depth information of transparent objects is often inaccurate. This leads to difficulties for robots in grasping transparent objects, as incorrect depth information can result in the robot being unable to predict or predict incorrectly the grasping pose. Therefore, it is necessary to complete the depth information for transparent objects. Previous methods for depth completion of transparent objects often struggle to balance accuracy and real-time performance simultaneously. To achieve this goal, in this paper, we propose a transparent object depth completion network called TCRNet based on a cascade refinement structure, which balances accuracy and real-time performance simultaneously. First, the network incorporates a cascade refinement structure in the decoding stage to refine features multiple times, improving the accuracy of depth information. Additionally, an attention module is designed to adjust the extracted features, enabling the network to focus on depth information features in transparent object regions. Finally, a transformer-based error module is implemented in the network’s final output stage to predict and adjust the error between the depth image and the ground truth. TCRNet is trained and tested on three datasets: ClearGrasp, Omniverse Object, and TransCG. It outperforms previous methods in terms of performance. Furthermore, TCRNet is applied to existing grasp detection methods to conduct grasping experiments on transparent objects using a real Baxter robot. Note to Practitioners —With the development of RGB-D camera technology, RGB-D cameras are now widely used in various scenarios such as industrial production, autonomous driving, and robot grasping. However, in certain situations where the camera faces transparent or highly reflective objects, the depth information captured by the camera is often not accurate enough, which can lead to subsequent accidents. Therefore, it is necessary to repair and complete the depth images to achieve accurate understanding of the scene’s depth information. In recent years, with the advancement of deep learning, deep learning-based depth image processing and restoration techniques have been widely applied. In this paper, we propose a high-accuracy network for repairing depth images of transparent objects, which can accurately restore and estimate the depth information of transparent objects in various scenarios. Moreover, experimental results demonstrate that our proposed method can generalize well to other unknown scenes, achieving excellent results.

TDCNet: Transparent Objects Depth Completion with CNN-Transformer Dual-Branch Parallel Network

TCRNet: Transparent Object Depth Completion with Cascade Refinements

A transformer-CNN parallel network for image guided depth completion

DFNet-Trans: An end-to-end multibranching network for depth estimation for transparent objects

TODE-Trans: Transparent Object Depth Estimation with Transformer

Transparent Depth Completion Using Segmentation Features

A Two-stage Hybrid CNN-Transformer Network for RGB Guided Indoor Depth Completion.

Transparent Object Depth Completion

TransCG: A Large-Scale Real-World Dataset for Transparent Object Depth Completion and a Grasping Baseline

RigNet++: Semantic Assisted Repetitive Image Guided Network for Depth Completion

TANet: Transformer-based Asymmetric Network for RGB-D Salient Object Detection

FDCT: Fast Depth Completion for Transparent Objects

VisionTwinNet: Gated Clarity Enhancement Paired With Light-Robust CD Transformers

DistillGrasp: Integrating Features Correlation with Knowledge Distillation for Depth Completion of Transparent Objects

Multi-Task Foreground-Aware Network with Depth Completion for Enhanced RGB-D Fusion Object Detection Based on Transformer

When CNN meet with ViT: decision-level feature fusion for camouflaged object detection

Transformer-Based Cross-Modal Integration Network for RGB-T Salient Object Detection

A Transformer-Based Image-Guided Depth-Completion Model with Dual-Attention Fusion Module

DBCTNet: Double Branch Convolution-Transformer Network for Hyperspectral Image Classification

Gated Cross-Attention Network for Depth Completion