Abstract:AbstractIn recent years, research into 3D shape recognition in the field of multimedia and computer vision has attracted wide attention. With the rapid development of deep learning, various deep models have achieved state-of-the-art performance based on different representations. There are many modalities for representing a 3D model, such as point cloud, multiview, and panorama view. Deep learning models based on these different modalities have different concerns, and all of them have achieved high performance for 3D shape recognition. However, all of these methods ignore the multimodality information in conditions where the same 3D model is represented by different modalities. Thus, we can obtain a better descriptor by guiding the training to consider these multiple representations. In this article, we propose MMFN, a novel multimodal fusion network for 3D shape recognition that employs correlations between the different modalities to generate a fused descriptor, which is more robust. In particular, we design two novel loss functions to help the model learn the correlation information during training. The first is correlation loss, which focuses on the correlations among different descriptors generated from different structures. This approach reduces the training time and improves the robustness of the fused descriptor of the 3D model. The second is instance loss, which preserves the independence of each modality and utilizes feature differentiation to guide model learning during the training process. More specifically, we use the weighted fusion method, which applies statistical methods to obtain robust descriptors that maximize the advantages of the information from the different modalities. We evaluated the proposed method on the ModelNet40 and ShapeNetCore55 datasets for 3D shape classification and retrieval tasks. The experimental results and comparisons with state-of-the-art methods demonstrate the superiority of our approach.

Multi-model Fusion Metric Learning for Image Set Classification

Multi-view Heterogeneous Fusion and Embedding for Categorical Attributes on Mixed Data.

Joint Dimensionality Reduction and Metric Learning for Image Set Classification

Multi-metric Learning for Multi-Sensor Fusion Based Classification.

Image classification with multi-view multi-instance metric learning

Multiple Riemannian Manifold-valued Descriptors based Image Set Classification with Multi-Kernel Metric Learning

MMFN: Multimodal Information Fusion Networks for 3D Model Classification and Retrieval

Multi-manifold Metric Learning for Face Recognition Based on Image Sets

Multi-view metric learning for visual classification

Multi-view Deep Metric Learning for Image Classification.

Multimodal Learning for View-Based 3D Object Classification

Image Set Classification Using Holistic Multiple Order Statistics Features and Localized Multi-kernel Metric Learning

Heterogeneous Multi-Metric Learning for Multi-Sensor Fusion.

Collaborative Multi-View Metric Learning For Visual Classification

Multi-modal Learning for Social Image Classification

Multi-Manifold Deep Metric Learning For Image Set Classification

Joint Metric Learning-Based Class-Specific Representation for Image Set Classification

Multi-scale Collaborative Coupled Metric Learning Method Based on the Fusion of Class and Structure Information

A Multi-View Fusion Method Via Tensor Learning And Gradient Descent For Image Features

Hierarchical Multi-View Metric Learning with HSIC Regularization

Equivariant Multi-Modality Image Fusion