LAM3D: Large Image-Point-Cloud Alignment Model for 3D Reconstruction from Single Image

Ruikai Cui,Xibin Song,Weixuan Sun,Senbo Wang,Weizhe Liu,Shenzhou Chen,Taizhang Shang,Yang Li,Nick Barnes,Hongdong Li,Pan Ji

2024-05-24

Abstract:Large Reconstruction Models have made significant strides in the realm of automated 3D content generation from single or multiple input images. Despite their success, these models often produce 3D meshes with geometric inaccuracies, stemming from the inherent challenges of deducing 3D shapes solely from image data. In this work, we introduce a novel framework, the Large Image and Point Cloud Alignment Model (LAM3D), which utilizes 3D point cloud data to enhance the fidelity of generated 3D meshes. Our methodology begins with the development of a point-cloud-based network that effectively generates precise and meaningful latent tri-planes, laying the groundwork for accurate 3D mesh reconstruction. Building upon this, our Image-Point-Cloud Feature Alignment technique processes a single input image, aligning to the latent tri-planes to imbue image features with robust 3D information. This process not only enriches the image features but also facilitates the production of high-fidelity 3D meshes without the need for multi-view input, significantly reducing geometric distortions. Our approach achieves state-of-the-art high-fidelity 3D mesh reconstruction from a single image in just 6 seconds, and experiments on various datasets demonstrate its effectiveness.

Computer Vision and Pattern Recognition

What problem does this paper attempt to address?

### Problems the Paper Attempts to Solve

The paper aims to address the issue of geometric distortion encountered when generating high-quality 3D mesh models from a single image. Although existing large reconstruction models (LRMs) have made significant progress in the automated generation of 3D content, these models often produce geometrically inaccurate 3D meshes due to the inherent challenges of inferring 3D shapes from image data.

Specifically, the paper proposes a new framework called **Large Image Point Cloud Alignment Model (LAM3D)**, which utilizes 3D point cloud data to enhance the fidelity of the generated 3D meshes. LAM3D achieves this goal through the following steps:

1. **Point Cloud Network Development**: First, a point cloud-based network is developed to generate precise and meaningful latent tri-planes, laying the foundation for accurate 3D mesh reconstruction.
2. **Image-Point Cloud Feature Alignment**: Then, through single input image processing techniques, the image features are aligned with the latent tri-planes, enriching the image features with robust 3D information. This process not only enhances the image features but also promotes the generation of high-fidelity 3D meshes without the need for multi-view inputs, significantly reducing geometric distortion.

Through these methods, LAM3D is capable of generating high-fidelity 3D meshes from a single image in just 6 seconds, and experiments on multiple datasets have validated its effectiveness.