Abstract:Human activities are inherently complex, often involving numerous object interactions. To better understand these activities, it is crucial to model their interactions with the environment captured through dynamic changes. The recent availability of affordable head-mounted cameras and egocentric data offers a more accessible and efficient means to understand human-object interactions in 3D environments. However, most existing methods for human activity modeling neglect the dynamic interactions with objects, resulting in only static representations. The few existing solutions often require inputs from multiple sources, including multi-camera setups, depth-sensing cameras, or kinesthetic sensors. To this end, we introduce EgoGaussian, the first method capable of simultaneously reconstructing 3D scenes and dynamically tracking 3D object motion from RGB egocentric input alone. We leverage the uniquely discrete nature of Gaussian Splatting and segment dynamic interactions from the background, with both having explicit representations. Our approach employs a clip-level online learning pipeline that leverages the dynamic nature of human activities, allowing us to reconstruct the temporal evolution of the scene in chronological order and track rigid object motion. EgoGaussian shows significant improvements in terms of both dynamic object and background reconstruction quality compared to the state-of-the-art. We also qualitatively demonstrate the high quality of the reconstructed models.

What problem does this paper attempt to address?

The problem that this paper attempts to solve is to reconstruct 3D scenes and the motion of dynamic objects from egocentric videos. Specifically, existing methods often overlook dynamic interactions when modeling human - object interactions, resulting in reconstruction results being limited to static representations and being prone to motion - induced artifacts (such as the "ghosting effect"). Moreover, most existing methods need to rely on complex devices such as multi - view inputs, depth - sensing cameras or kinesthetic sensors, and cannot effectively reconstruct dynamic scenes from RGB videos alone. To solve these problems, the paper proposes a new method named EgoGaussian, which aims to simultaneously reconstruct 3D scenes and track the dynamic motion of 3D objects from a single RGB first - person video. Its core contributions include: 1. **Dynamic scene reconstruction**: By distinguishing between static and dynamic segments in the video, EgoGaussian can accurately reconstruct the shapes of the background and dynamic objects and track their motion. 2. **Utilizing Gaussian Splatting**: This method takes advantage of the unique discrete nature of 3D Gaussian splatting, which can clearly distinguish between foreground objects and the background, thereby achieving high - quality 4D reconstruction. 3. **Online learning framework**: EgoGaussian adopts a segment - based online learning framework, which can gradually reconstruct the temporal evolution process of the scene according to the dynamic characteristics of human activities and track the motion of rigid objects. ### Specific problems and solutions - **Problem**: Existing methods cannot reconstruct dynamic scenes from a single RGB video alone. - **Solution**: EgoGaussian achieves high - quality reconstruction of dynamic objects and the background by segmenting the video into static and dynamic segments and using Gaussian Splatting for modeling. - **Problem**: Existing methods perform poorly when dealing with dynamic interactions and are prone to produce artifacts. - **Solution**: EgoGaussian avoids motion - induced artifacts by clearly distinguishing between dynamic and static parts and combining hand - object segmentation masks. - **Problem**: Existing methods rely on multi - view inputs or other complex devices. - **Solution**: EgoGaussian only requires a single RGB video as input, reducing hardware requirements and making dynamic scene reconstruction more convenient and efficient. ### Experimental verification The paper conducts experimental verification through two publicly available first - person video datasets, HOI4D and EPIC - KITCHENS, demonstrating the superior performance of EgoGaussian in the new view synthesis task. The experimental results show that EgoGaussian significantly outperforms the existing state - of - the - art methods in evaluation metrics such as PSNR, SSIM, and LPIPS. In conclusion, EgoGaussian solves the deficiencies of existing technologies in dynamic scene reconstruction through innovative methods and provides a more efficient and accurate solution.

EgoGaussian: Dynamic Scene Understanding from Egocentric Video with 3D Gaussian Splatting

EF-3DGS: Event-Aided Free-Trajectory 3D Gaussian Splatting

Ego3DT: Tracking Every 3D Object in Ego-centric Videos

Event3DGS: Event-Based 3D Gaussian Splatting for High-Speed Robot Egomotion

EgoLifter: Open-world 3D Segmentation for Egocentric Perception

3D-Aware Instance Segmentation and Tracking in Egocentric Videos

Dynamic 3D Gaussian Tracking for Graph-Based Neural Dynamics Modeling

4D Human Body Capture from Egocentric Video via 3D Scene Grounding

EVA-Gaussian: 3D Gaussian-based Real-time Human Novel View Synthesis under Diverse Camera Settings

EvGGS: A Collaborative Learning Framework for Event-based Generalizable Gaussian Splatting

Improving Dynamic 3D Gaussian Splatting from Monocular Videos with Object Motion Information

Real-time Photorealistic Dynamic Scene Representation and Rendering with 4D Gaussian Splatting

Motion-aware 3D Gaussian Splatting for Efficient Dynamic Scene Reconstruction

Object-centric Reconstruction and Tracking of Dynamic Unknown Objects using 3D Gaussian Splatting

GaussianPrediction: Dynamic 3D Gaussian Prediction for Motion Extrapolation and Free View Synthesis

Guess The Unseen: Dynamic 3D Scene Reconstruction from Partial 2D Glimpses

MotionGS: Exploring Explicit Motion Guidance for Deformable 3D Gaussian Splatting

Gaussian Splatting LK

Scene-aware Egocentric 3D Human Pose Estimation

Omni-Scene: Omni-Gaussian Representation for Ego-Centric Sparse-View Scene Reconstruction