Abstract:We introduce NoPoSplat, a feed-forward model capable of reconstructing 3D scenes parameterized by 3D Gaussians from \textit{unposed} sparse multi-view images. Our model, trained exclusively with photometric loss, achieves real-time 3D Gaussian reconstruction during inference. To eliminate the need for accurate pose input during reconstruction, we anchor one input view's local camera coordinates as the canonical space and train the network to predict Gaussian primitives for all views within this space. This approach obviates the need to transform Gaussian primitives from local coordinates into a global coordinate system, thus avoiding errors associated with per-frame Gaussians and pose estimation. To resolve scale ambiguity, we design and compare various intrinsic embedding methods, ultimately opting to convert camera intrinsics into a token embedding and concatenate it with image tokens as input to the model, enabling accurate scene scale prediction. We utilize the reconstructed 3D Gaussians for novel view synthesis and pose estimation tasks and propose a two-stage coarse-to-fine pipeline for accurate pose estimation. Experimental results demonstrate that our pose-free approach can achieve superior novel view synthesis quality compared to pose-required methods, particularly in scenarios with limited input image overlap. For pose estimation, our method, trained without ground truth depth or explicit matching loss, significantly outperforms the state-of-the-art methods with substantial improvements. This work makes significant advances in pose-free generalizable 3D reconstruction and demonstrates its applicability to real-world scenarios. Code and trained models are available at <a class="link-external link-https" href="https://noposplat.github.io/" rel="external noopener nofollow">this https URL</a>.

A Pose-only Solution to Visual Reconstruction and Navigation

A Pose-only Solution to Visual Reconstruction and Navigation

Pose Estimation and Neural Implicit Reconstruction Towards Non-Cooperative Spacecraft Without Offline Prior Information

Visual Odometry Based 3D-Reconstruction

Direct Alignment with Generalized Correspondences: A Unified Framework for Structure-Based Visual Pose Estimation.

Structure and Pose from Single Images of Symmetric Objects with Applications to Robot Navigation

Robust 3D Reconstruction with an RGB-D Camera

Equivalent Constraints for Two-View Geometry: Pose Solution/Pure Rotation Identification and 3D Reconstruction

Rethinking Reprojection: Closing the Loop for Pose-aware ShapeReconstruction from a Single Image

Reconstructing Interactive 3D Scenes by Panoptic Mapping and CAD Model Alignments

High-Accuracy Pose Estimation for 3D Reconstruction Considering the Manipulator Motion Constraints with an Eye-in-Hand RGB-D Camera

Free-Moving Object Reconstruction and Pose Estimation with Virtual Camera

Real-time 3D Scene Reconstruction with Dynamically Moving Object Using a Single Depth Camera

Active Scene Reconstruction Via Self-Rotation Driven by Optimized Information Theory

A Highly Robust Automatic 3D Reconstruction System Based on Integrated Optimization by Point Line Features

Scene Reconstruction with Functional Objects for Robot Autonomy

A 3D reconstruction algorithm of multiple objects

3D Reconstruction of Indoor Scenes Based on Feature and Graph Optimization

No Pose, No Problem: Surprisingly Simple 3D Gaussian Splats from Sparse Unposed Images

Know Your Neighbors: Improving Single-View Reconstruction via Spatial Vision-Language Reasoning

3D Reconstruction for Robot Navigation Based on Projection of Virtual Height Line and Its Performance Evaluation