Abstract:We propose a novel approach for 3D mesh reconstruction from multi-view images. Our method takes inspiration from large reconstruction models like LRM that use a transformer-based triplane generator and a Neural Radiance Field (NeRF) model trained on multi-view images. However, in our method, we introduce several important modifications that allow us to significantly enhance 3D reconstruction quality. First of all, we examine the original LRM architecture and find several shortcomings. Subsequently, we introduce respective modifications to the LRM architecture, which lead to improved multi-view image representation and more computationally efficient training. Second, in order to improve geometry reconstruction and enable supervision at full image resolution, we extract meshes from the NeRF field in a differentiable manner and fine-tune the NeRF model through mesh rendering. These modifications allow us to achieve state-of-the-art performance on both 2D and 3D evaluation metrics, such as a PSNR of 28.67 on Google Scanned Objects (GSO) dataset. Despite these superior results, our feed-forward model still struggles to reconstruct complex textures, such as text and portraits on assets. To address this, we introduce a lightweight per-instance texture refinement procedure. This procedure fine-tunes the triplane representation and the NeRF color estimation model on the mesh surface using the input multi-view images in just 4 seconds. This refinement improves the PSNR to 29.79 and achieves faithful reconstruction of complex textures, such as text. Additionally, our approach enables various downstream applications, including text- or image-to-3D generation.

What problem does this paper attempt to address?

### What problems does this paper attempt to solve? This paper aims to solve the geometric and texture reconstruction problems encountered when generating high - quality 3D mesh models from multi - view images. Specifically, the authors propose a new method to improve the large - scale 3D reconstruction model (LRM). By introducing geometric and texture optimization techniques, the quality of 3D reconstruction is significantly improved. #### Main problems: 1. **Poor geometric reconstruction quality**: Existing large - scale 3D reconstruction models (such as LRM) have difficulty in generating high - quality geometric structures when using the Marching Cube (MC) algorithm to extract meshes, and are prone to mesh - shaped artifacts. 2. **Inaccurate texture reconstruction**: Existing methods perform poorly in reconstructing complex textures (such as text, portraits, etc.), especially when dealing with high - resolution images. 3. **Low computational efficiency**: Existing methods have high computational costs during training and inference, resulting in slow speeds. #### Solutions: 1. **Improve the LRM architecture**: - **Replace Transformer with convolutional encoder**: Replace the pre - trained DiNO Transformer with a convolutional encoder to preserve more local details. - **Replace the up - sampling layer**: Replace the traditional deconvolution layer with a Pixelshuffle layer to reduce mesh - shaped artifacts. - **Separate the prediction of density and color**: Use two shallow MLPs to predict density and color respectively for easy subsequent fine - tuning. 2. **Geometric optimization**: - **NeRF initialization and mesh rendering fine - tuning**: First, train through NeRF volume rendering, and then use differentiable Marching Cubes (DiffMC) to extract meshes and perform mesh - rendering fine - tuning, thereby achieving full - resolution image supervision. 3. **Texture optimization**: - **Light - weight instance - by - instance texture optimization**: Quickly improve the texture quality by fine - tuning the tri - plane representation and color estimation model for each instance, and the optimization can be completed in only 4 seconds. Through these improvements, the experimental results of this method on the Google Scanned Objects (GSO) and OmniObject3D datasets show that it outperforms existing methods in both 2D and 3D evaluation metrics, especially in metrics such as PSNR, SSIM, LPIPS, CD, and IoU. In addition, this method can also be applied to various downstream tasks, such as text / image - to - 3D generation, further expanding its application range.

GTR: Improving Large 3D Reconstruction Models through Geometry and Texture Refinement

MeshLRM: Large Reconstruction Model for High-Quality Mesh

$R^2$-Mesh: Reinforcement Learning Powered Mesh Reconstruction via Geometry and Appearance Refinement

GeoLRM: Geometry-Aware Large Reconstruction Model for High-Quality 3D Gaussian Generation

GRM: Large Gaussian Reconstruction Model for Efficient 3D Reconstruction and Generation

CRM: Single Image to 3D Textured Mesh with Convolutional Reconstruction Model

2L3: Lifting Imperfect Generated 2D Images into Accurate 3D

Single-view 3D Scene Reconstruction with High-fidelity Shape and Texture

GS-LRM: Large Reconstruction Model for 3D Gaussian Splatting

Multi-View Large Reconstruction Model via Geometry-Aware Positional Encoding and Attention

Triplane Meets Gaussian Splatting: Fast and Generalizable Single-View 3D Reconstruction with Transformers

MeshFormer: High-Quality Mesh Generation with 3D-Guided Reconstruction Model

Real-Time Globally Consistent Dense 3D Reconstruction with Online Texturing.

Enhance-NeRF: Multiple Performance Evaluation for Neural Radiance Fields

Omni-Recon: Harnessing Image-based Rendering for General-Purpose Neural Radiance Fields

Georecon: a coarse-to-fine visual 3D reconstruction approach for high-resolution images with neural matching priors

FineRecon: Depth-aware Feed-forward Network for Detailed 3D Reconstruction

ReTR: Modeling Rendering Via Transformer for Generalizable Neural Surface Reconstruction

G3R: Gradient Guided Generalizable Reconstruction

High-Quality Textured 3D Shape Reconstruction with Cascaded Fully Convolutional Networks