Abstract:AbstractIn recent years, research into 3D shape recognition in the field of multimedia and computer vision has attracted wide attention. With the rapid development of deep learning, various deep models have achieved state-of-the-art performance based on different representations. There are many modalities for representing a 3D model, such as point cloud, multiview, and panorama view. Deep learning models based on these different modalities have different concerns, and all of them have achieved high performance for 3D shape recognition. However, all of these methods ignore the multimodality information in conditions where the same 3D model is represented by different modalities. Thus, we can obtain a better descriptor by guiding the training to consider these multiple representations. In this article, we propose MMFN, a novel multimodal fusion network for 3D shape recognition that employs correlations between the different modalities to generate a fused descriptor, which is more robust. In particular, we design two novel loss functions to help the model learn the correlation information during training. The first is correlation loss, which focuses on the correlations among different descriptors generated from different structures. This approach reduces the training time and improves the robustness of the fused descriptor of the 3D model. The second is instance loss, which preserves the independence of each modality and utilizes feature differentiation to guide model learning during the training process. More specifically, we use the weighted fusion method, which applies statistical methods to obtain robust descriptors that maximize the advantages of the information from the different modalities. We evaluated the proposed method on the ModelNet40 and ShapeNetCore55 datasets for 3D shape classification and retrieval tasks. The experimental results and comparisons with state-of-the-art methods demonstrate the superiority of our approach.

Multi-view Moments Embedding Network for 3D Shape Recognition

MANet: Multimodal Attention Network based Point- View fusion for 3D Shape Recognition

Learning Multi-View Representation with LSTM for 3-D Shape Recognition and Retrieval

Multi-view Fusion with Deep Learning for 3D Shape Classification

PVNet: A Joint Convolutional Network of Point Cloud and Multi-View for 3D Shape Recognition.

Learning Disentangled Representation for Multi-View 3D Object Recognition.

MVPN: Multi-View Prototype Network for 3D Shape Recognition

Multi-View CNN Feature Aggregation with ELM Auto-Encoder for 3D Shape Recognition

MV-LFN: Multi-view Based Local Information Fusion Network for 3D Shape Recognition

Attention-Guided Fusion Network of Point Cloud and Multiple Views for 3D Shape Recognition.

MHSAN: Multi-view hierarchical self-attention network for 3D shape recognition

HRGE-Net: Hierarchical Relational Graph Embedding Network for Multi-view 3D Shape Recognition

PVRNet: Point-View Relation Neural Network for 3D Shape Recognition

Multi-view Convolutional Neural Networks for 3D Shape Recognition

Hamming Embedding Sensitivity Guided Fusion Network for 3D Shape Representation.

MVCLN: Multi-View Convolutional LSTM Network for Cross-Media 3D Shape Recognition

Deep Learning Multi-View Representation for Face Recognition

Learning Discriminative and Generative Shape Embeddings for Three-Dimensional Shape Retrieval

Learning View-Based Graph Convolutional Network for Multi-View 3D Shape Analysis

SVHAN: Sequential View Based Hierarchical Attention Network for 3D Shape Recognition.

MMFN: Multimodal Information Fusion Networks for 3D Model Classification and Retrieval