CLN: Cross-Domain Learning Network for 2D Image-Based 3D Shape Retrieval

Weizhi Nie,Yue Zhao,Jie Nie,An-An Liu,Sicheng Zhao

DOI: https://doi.org/10.1109/tcsvt.2021.3070969

IF: 5.859

2022-03-01

IEEE Transactions on Circuits and Systems for Video Technology

Abstract:Retrieving 3D shapes based on 2D images is a challenging research topic, due to the significant gap between different domains. Recently, various approaches have been proposed to handle this problem. However, the majority of methods target the cross-domain retrieval task as a pure domain adaptation problem, which focuses on the alignment but ignores the visual relevance between the 2D images and their corresponding 3D shapes. To fundamentally decrease the divergence between different domains, we propose a novel cross-domain learning network (CLN) for 2D image-based 3D shape retrieval task. First, we estimate the pose information from the 2D image to guide the view rendering of 3D shapes, which increases the visual correlations of the cross-domain data to eliminate the divergence between them. Second, we introduce a novel joint learning network, considering both the domain-specific characteristics and the cross-domain interactions for data alignment, which further compensates for the gap between different domains by controlling the distance of intra- and inter-classes. After the metric learning process, discriminative descriptors of images and shapes are generated for the cross-domain retrieval task. To prove the effectiveness and robustness of the proposed method, we conduct extensive experiments on the MI3DOR, SHREC'13, and SHREC'14 datasets. The experimental results demonstrate the superiority of our proposed method, and significant improvements have been achieved compared with state-of-the-art methods.

engineering, electrical & electronic

What problem does this paper attempt to address?

The problem this paper attempts to address is the significant domain gap that exists when performing 3D shape retrieval based on 2D images. Specifically, due to the substantial differences in data representation and distribution between 2D images and 3D shapes, cross-domain retrieval tasks are highly challenging. Existing methods mainly focus on the domain adaptation problem by aligning features from different domains to reduce the domain gap, but they often overlook the visual correlation between 2D images and their corresponding 3D shapes.

To fundamentally reduce the differences between different domains, the authors propose a new Cross-domain Learning Network (CLN) for 2D image-based 3D shape retrieval tasks. The main contributions of this method include:

1. **Introduction of a Pose Estimation Network**: By predicting the pose information of 2D images as an effective prior for 3D shape representation, it increases the visual correlation between different domains, effectively compensating for the domain gap.
2. **Joint Learning Network**: Considering intra-domain characteristics and cross-domain interactions, it reduces the gap between different domains by controlling intra-class similarity and inter-class distinctiveness.
3. **Experimental Validation**: Extensive experiments were conducted on three popular datasets: MI3DOR, SHREC’13, and SHREC’14. The results show that this method outperforms other existing methods.

In summary, this paper aims to improve the performance and robustness of 2D image-based 3D shape retrieval tasks by introducing pose estimation and a joint learning network.