Abstract:This paper presents GGRt, a novel approach to generalizable novel viewsynthesis that alleviates the need for real camera poses, complexity inprocessing high-resolution images, and lengthy optimization processes, thusfacilitating stronger applicability of 3D Gaussian Splatting (3D-GS) inreal-world scenarios. Specifically, we design a novel joint learning frameworkthat consists of an Iterative Pose Optimization Network (IPO-Net) and aGeneralizable 3D-Gaussians (G-3DG) model. With the joint learning mechanism,the proposed framework can inherently estimate robust relative pose informationfrom the image observations and thus primarily alleviate the requirement ofreal camera poses. Moreover, we implement a deferred back-propagation mechanismthat enables high-resolution training and inference, overcoming the resolutionconstraints of previous methods. To enhance the speed and efficiency, wefurther introduce a progressive Gaussian cache module that dynamically adjustsduring training and inference. As the first pose-free generalizable 3D-GSframework, GGRt achieves inference at ≥ 5 FPS and real-time rendering at≥ 100 FPS. Through extensive experimentation, we demonstrate that ourmethod outperforms existing NeRF-based pose-free techniques in terms ofinference speed and effectiveness. It can also approach the real pose-based3D-GS methods. Our contributions provide a significant leap forward for theintegration of computer vision and computer graphics into practicalapplications, offering state-of-the-art results on LLFF, KITTI, and Waymo Opendatasets and enabling real-time rendering for immersive experiences.

SplatSim: Zero-Shot Sim2Real Transfer of RGB Manipulation Policies Using Gaussian Splatting

RL-GSBridge: 3D Gaussian Splatting Based Real2Sim2Real Method for Robotic Manipulation Learning

RoboGSim: A Real2Sim2Real Robotic Gaussian Splatting Simulator

Gaussian Splatting to Real World Flight Navigation Transfer with Liquid Networks

AutoSplat: Constrained Gaussian Splatting for Autonomous Driving Scene Reconstruction

Robotic Learning in your Backyard: A Neural Simulator from Open Source Components

SplatAD: Real-Time Lidar and Camera Rendering with 3D Gaussian Splatting for Autonomous Driving

Close the Sim2real Gap via Physically-based Structured Light Synthetic Data Simulation

Let's Make a Splan: Risk-Aware Trajectory Optimization in a Normalized Gaussian Splat

SOUS VIDE: Cooking Visual Drone Navigation Policies in a Gaussian Splatting Vacuum

Splatt3R: Zero-shot Gaussian Splatting from Uncalibrated Image Pairs

Auto-Tuned Sim-to-Real Transfer

Splat-MOVER: Multi-Stage, Open-Vocabulary Robotic Manipulation via Editable Gaussian Splatting

Physically Embodied Gaussian Splatting: A Realtime Correctable World Model for Robotics

GraspSplats: Efficient Manipulation with 3D Feature Splatting

Splatfacto-W: A Nerfstudio Implementation of Gaussian Splatting for Unconstrained Photo Collections

Local Policies Enable Zero-shot Long-horizon Manipulation

MGSO: Monocular Real-time Photometric SLAM with Efficient 3D Gaussian Splatting

SAFER-Splat: A Control Barrier Function for Safe Navigation with Online Gaussian Splatting Maps

RaSim: A Range-aware High-fidelity RGB-D Data Simulation Pipeline for Real-world Applications

GGRt: Towards Pose-free Generalizable 3D Gaussian Splatting in Real-time