GeoLRM: Geometry-Aware Large Reconstruction Model for High-Quality 3D Gaussian Generation

Chubin Zhang,Hongliang Song,Yi Wei,Yu Chen,Jiwen Lu,Yansong Tang

2024-10-28

Abstract:In this work, we introduce the Geometry-Aware Large Reconstruction Model (GeoLRM), an approach which can predict high-quality assets with 512k Gaussians and 21 input images in only 11 GB GPU memory. Previous works neglect the inherent sparsity of 3D structure and do not utilize explicit geometric relationships between 3D and 2D images. This limits these methods to a low-resolution representation and makes it difficult to scale up to the dense views for better quality. GeoLRM tackles these issues by incorporating a novel 3D-aware transformer structure that directly processes 3D points and uses deformable cross-attention mechanisms to effectively integrate image features into 3D representations. We implement this solution through a two-stage pipeline: initially, a lightweight proposal network generates a sparse set of 3D anchor points from the posed image inputs; subsequently, a specialized reconstruction transformer refines the geometry and retrieves textural details. Extensive experimental results demonstrate that GeoLRM significantly outperforms existing models, especially for dense view inputs. We also demonstrate the practical applicability of our model with 3D generation tasks, showcasing its versatility and potential for broader adoption in real-world applications. The project page: <a class="link-external link-https" href="https://linshan-bin.github.io/GeoLRM/" rel="external noopener nofollow">this https URL</a>.

Computer Vision and Pattern Recognition

What problem does this paper attempt to address?

The problem that this paper attempts to solve is: Existing 3D asset generation methods have limitations when dealing with high - resolution and multi - view inputs. Specifically, traditional methods overlook the inherent sparsity of 3D structures and do not fully utilize the explicit geometric relationships between 3D and 2D images, which makes these methods difficult to scale to higher - quality dense - view inputs. These problems limit the quality and diversity of 3D assets, especially in applications that require efficient generation of high - resolution 3D models, such as in fields like robotics and virtual reality.

To solve these problems, the paper proposes a method named GeoLRM (Geometry - Aware Large Reconstruction Model). This method predicts high - quality 3D Gaussian distributions by introducing a new 3D - aware Transformer structure and can effectively integrate image features into 3D representations. GeoLRM achieves this goal through a two - stage process: First, a lightweight proposal network generates a set of sparse 3D anchor points from the input images; then, a specialized reconstruction Transformer refines the geometric structure and restores texture details. This method not only significantly improves the generation quality of 3D assets but also can maintain high performance with more input views, thus providing greater flexibility and potential for practical applications.