Abstract:In recent years, 3D understanding has turned to 2D vision-language pre-trained models to overcome data scarcity challenges. However, existing methods simply transfer 2D alignment strategies, aligning 3D representations with single-view 2D images and coarse-grained parent category text. These approaches introduce information degradation and insufficient synergy issues, leading to performance loss. Information degradation arises from overlooking the fact that a 3D representation should be equivalent to a series of multi-view images and more fine-grained subcategory text. Insufficient synergy neglects the idea that a robust 3D representation should align with the joint vision-language space, rather than independently aligning with each modality. In this paper, we propose a multi-view joint modality modeling approach, termed JM3D, to obtain a unified representation for point cloud, text, and image. Specifically, a novel Structured Multimodal Organizer (SMO) is proposed to address the information degradation issue, which introduces contiguous multi-view images and hierarchical text to enrich the representation of vision and language modalities. A Joint Multi-modal Alignment (JMA) is designed to tackle the insufficient synergy problem, which models the joint modality by incorporating language knowledge into the visual modality. Extensive experiments on ModelNet40 and ScanObjectNN demonstrate the effectiveness of our proposed method, JM3D, which achieves state-of-the-art performance in zero-shot 3D classification. JM3D outperforms ULIP by approximately 4.3% on PointMLP and achieves an improvement of up to 6.5% accuracy on PointNet++ in top-1 accuracy for zero-shot 3D classification on ModelNet40. The source code and trained models for all our experiments are publicly available at <a class="link-external link-https" href="https://github.com/Mr-Neko/JM3D" rel="external noopener nofollow">this https URL</a>.

QSCC: A Quaternion Semantic Cell Convolution Graph Neural Network for MBD Product Model Classification

MoDL-QSM: Model-based Deep Learning for Quantitative Susceptibility Mapping

Target-oriented Sentiment Classification with Sequential Cross-modal Semantic Graph

Low-Voltage Electrical Product Quality Problem-Solving Based on Improved Deep Structured Semantic Model.

3D Model Classification Based on GCN and SVM

Fast Quaternion Product Units for Learning Disentangled Representations in SO(3)

Multilevel Cross-Aware RGBD Indoor Semantic Segmentation for Bionic Binocular Robot

Simulation of Grey Neural Network in MCMQC

A Semi-supervised Multi-channel Graph Convolutional Network for Query Classification in E-commerce

A Deep Learning Approach to the Classification of 3D CAD Models

Global Clue-Guided Cross-Memory Quaternion Transformer Network for Multisource Remote Sensing Data Classification

DeepQMLP: A Scalable Quantum-Classical Hybrid DeepNeural Network Architecture for Classification

Multitask Deep Learning with Dynamic Task Balancing for Quantum Mechanical Properties Prediction

Merging Subject Matter Expertise and Deep Convolutional Neural Network for State-Based Online Machine-Part Interaction Classification

Beyond First Impressions: Integrating Joint Multi-modal Cues for Comprehensive 3D Representation

3D convolutional neural network for machining feature recognition with gradient-based visual explanations from 3D CAD models

Quantum mesh neural network model in precise image diagnosing

QGD-Net: A Lightweight Model Utilizing Pixels of Affinity in Feature Layer for Dermoscopic Lesion Segmentation

Voxel-based three-view hybrid parallel network for 3D object classification

Quaternion-based machine learning on topological quantum systems

Automatic Fine-Grained BIM element classification using Multi-Modal deep learning (MMDL)