Toward Large-Scale Plenoptic Reconstruction

Lu Fang

DOI: https://doi.org/10.1007/978-981-97-6915-5_5

2024-01-01

Abstract:AbstractReconstructing real-world scenes with unparalleled levels of realism and detail has been a long-standing goal in the fields of computer vision and graphics. Achieving this goal necessitates coordinated efforts in both sensing techniques and plenoptic reconstruction algorithms.

What problem does this paper attempt to address?

Plenoptic Reconstruction

Lu Fang

DOI: https://doi.org/10.1007/978-981-97-6915-5_4

2024-01-01

Abstract:AbstractEmpowered by advanced plenoptic sensing systems, light-field imaging becomes one of the most extensively used methods for capturing 3D views of a scene. In contrast to the traditional input to a 3D graphics system, namely, scenes consisting of pre-defined geometric primitives with different materials and sets of lights, the input to a light field is only a set of 2D images which are informative and cost effective. Unfortunately, due to the limited sensor resolution, existing systems must balance the spatial and angular resolution, i.e., one can obtain dense sampling images in the spatial dimension but only sparse sampling images in the angular (viewing angle) dimension or vice versa.
Theory and Approach of Large-Scale Computational Reconstruction

Bian Liheng,Li Daoyu,Chang Xuyang,Suo Jinli

DOI: https://doi.org/10.3788/lop221245

2023-01-01

Laser & Optoelectronics Progress

Abstract:Computational imaging compressively encodes high-dimensional scene data into low-dimensional measurements and recovers the high-dimensional scene information using computational reconstruction techniques. In the era of big data, the increasing demands for high spatiotemporal resolution have promoted the development of large-scale reconstruction algorithms with high accuracy, low complexity, and flexibility for various imaging systems. The existing large- scale computational reconstruction methods, including alternating projection, deep image prior, and plug- and-play optimization methods, have made great progresses over the past decades. Among the abovementioned methods, the alternating projection has been utilized in gigapixel quantitative phase imaging systems. Besides, the deep image prior and plug- and-play optimization techniques combine the advantages of conventional optimization and deep learning, which hold great potential for large-scale reconstruction. This work reviews the architectures and applications of these methods and prospects for the research trends, which can provide highlights for future works of large- scale computational imaging.
Large-Scale 3D Reconstruction from Multi-View Imagery: A Comprehensive Review

Haitao Luo,Jinming Zhang,Xiongfei Liu,Lili Zhang,Junyi Liu

DOI: https://doi.org/10.3390/rs16050773

IF: 5

2024-02-23

Remote Sensing

Abstract:Three-dimensional reconstruction is a key technology employed to represent virtual reality in the real world, which is valuable in computer vision. Large-scale 3D models have broad application prospects in the fields of smart cities, navigation, virtual tourism, disaster warning, and search-and-rescue missions. Unfortunately, most image-based studies currently prioritize the speed and accuracy of 3D reconstruction in indoor scenes. While there are some studies that address large-scale scenes, there has been a lack of systematic comprehensive efforts to bring together the advancements made in the field of 3D reconstruction in large-scale scenes. Hence, this paper presents a comprehensive overview of a 3D reconstruction technique that utilizes multi-view imagery from large-scale scenes. In this article, a comprehensive summary and analysis of vision-based 3D reconstruction technology for large-scale scenes are presented. The 3D reconstruction algorithms are extensively categorized into traditional and learning-based methods. Furthermore, these methods can be categorized based on whether the sensor actively illuminates objects with light sources, resulting in two categories: active and passive methods. Two active methods, namely, structured light and laser scanning, are briefly introduced. The focus then shifts to structure from motion (SfM), stereo matching, and multi-view stereo (MVS), encompassing both traditional and learning-based approaches. Additionally, a novel approach of neural-radiance-field-based 3D reconstruction is introduced. The workflow and improvements in large-scale scenes are elaborated upon. Subsequently, some well-known datasets and evaluation metrics for various 3D reconstruction tasks are introduced. Lastly, a summary of the challenges encountered in the application of 3D reconstruction technology in large-scale outdoor scenes is provided, along with predictions for future trends in development.

environmental sciences,imaging science & photographic technology,remote sensing,geosciences, multidisciplinary
DP-MVS: Detail Preserving Multi-View Surface Reconstruction of Large-Scale Scenes

Liyang Zhou,Zhuang Zhang,Hanqing Jiang,Han Sun,Hujun Bao,Guofeng Zhang

DOI: https://doi.org/10.3390/rs13224569

IF: 5

2021-01-01

Remote Sensing

Abstract:This paper presents an accurate and robust dense 3D reconstruction system for detail preserving surface modeling of large-scale scenes from multi-view images, which we named DP-MVS. Our system performs high-quality large-scale dense reconstruction, which preserves geometric details for thin structures, especially for linear objects. Our framework begins with a sparse reconstruction carried out by an incremental Structure-from-Motion. Based on the reconstructed sparse map, a novel detail preserving PatchMatch approach is applied for depth estimation of each image view. The estimated depth maps of multiple views are then fused to a dense point cloud in a memory-efficient way, followed by a detail-aware surface meshing method to extract the final surface mesh of the captured scene. Experiments on ETH3D benchmark show that the proposed method outperforms other state-of-the-art methods on F1-score, with the running time more than 4 times faster. More experiments on large-scale photo collections demonstrate the effectiveness of the proposed framework for large-scale scene reconstruction in terms of accuracy, completeness, memory saving, and time efficiency.
Outdoor Scene 3D Reconstruction from Multiple Point Cloud

Chen Hui,Feng Yan,Sun Jinzhong,Yang Jian,Cui Chenggang

DOI: https://doi.org/10.1117/12.2524421

2019-01-01

Abstract:This paper presents a 3D reconstruction method fusing structure from motion and laser scanning method. The precision of the 3D reconstruction is improved by combining point cloud obtained from different sensors. First, a scaled principal component analysis-iterative closest point (a scaled-PCA-ICP) algorithm is proposed to do the registration that can overcome the influence of large scale variance. Then, with the large scale factor, a recalculated registration center is proposed by region segmentation to realize the point cloud registration again. Finally, the two-view point clouds are robustly matched using the proposed optimization method to complete 3D color reconstruction of the outdoor large scenes. The proposed 3D reconstruction method is evaluated on the non-synchronous database of the real-world multi-view vision sequences obtained in an outdoor environment and laser scanner. The experiment results show improvements in both accuracy and efficiency.
GigaVision: when Computer Vision Meets Gigapixel Videography

Lu Fang

DOI: https://doi.org/10.1007/978-981-97-6915-5_6

2024-01-01

Abstract:AbstractIn previous chapters, we have explored advanced plenoptic imaging and reconstruction techniques, enabling images and videos to reach gigapixel-level resolution. This breakthrough unlocks new possibilities for a wide range of applications and industries. However, traditional computer vision methods, tailored for megapixel-level data, are ill-equipped to handle the complexities of gigapixel-level data, which often feature large-scale scenes with hundreds of objects and intricate interactions. As a result, these methods face significant limitations in both precision and efficiency.
A Divide-and-conquer Approach to Large Scene Reconstruction with Interactive Scene Analysis and Segmentation

Bo Jiang,Xinguo Liu

DOI: https://doi.org/10.1145/2534329.2534372

2013-01-01

Abstract:3D reconstruction of real world objects is a fundamental problem in computer vision and computer graphics. It is a challenge for high fidelity reconstruction with consumer-grade depth camera, e.g. Microsoft Kinect on large scale scene. A divide-and-conquer approach could both preserve local geometry details of scene objects and global consistency of the whole scene, thus is superior to existing GPU based real-time/global optimization based offline methods. In this paper, we extend the state-of-the-art in two folds. First, more fine grit scene segmentation is conducted for better local details preservation. Second, a scene analysis procedure is proposed for more reasonable global error spread. Preliminary experiments show that our method achieves more convincing reconstruction results.
VastGaussian: Vast 3D Gaussians for Large Scene Reconstruction

Jiaqi Lin,Zhihao Li,Xiao Tang,Jianzhuang Liu,Shiyong Liu,Jiayue Liu,Yangdi Lu,Xiaofei Wu,Songcen Xu,Youliang Yan,Wenming Yang

2024-02-27

Abstract:Existing NeRF-based methods for large scene reconstruction often have limitations in visual quality and rendering speed. While the recent 3D Gaussian Splatting works well on small-scale and object-centric scenes, scaling it up to large scenes poses challenges due to limited video memory, long optimization time, and noticeable appearance variations. To address these challenges, we present VastGaussian, the first method for high-quality reconstruction and real-time rendering on large scenes based on 3D Gaussian Splatting. We propose a progressive partitioning strategy to divide a large scene into multiple cells, where the training cameras and point cloud are properly distributed with an airspace-aware visibility criterion. These cells are merged into a complete scene after parallel optimization. We also introduce decoupled appearance modeling into the optimization process to reduce appearance variations in the rendered images. Our approach outperforms existing NeRF-based methods and achieves state-of-the-art results on multiple large scene datasets, enabling fast optimization and high-fidelity real-time rendering.

Computer Vision and Pattern Recognition
High-Resolution Plenoptic Sensing

Lu Fang

DOI: https://doi.org/10.1007/978-981-97-6915-5_3

2024-01-01

Abstract:AbstractThe high-resolution plenoptic sensing system can capture light field with a high level of detail and clarity. Resolution refers to the amount of detail that can be discerned in the reconstructed image or light field. The cutting edge of high-resolution plenoptic sensing is gigapixel plenoptic sensing, specifically refers to a system capable of capturing images with billions of pixels, resulting in extremely high-resolution light field.
A New Era of Indoor Scene Reconstruction: A Survey

Hao Wang,Minghui Li

DOI: https://doi.org/10.1109/access.2024.3440260

IF: 3.9

2024-08-18

IEEE Access

Abstract:Indoor scene reconstruction plays a pivotal role in the fields of computer graphics and vision, significantly impacting areas such as robotics and augmented reality. Recently, advancements in Neural Radiance Fields (NeRF) and 3D Gaussian Splashing (3DGS) technologies have notably enhanced scene rendering efficiency, accelerated the generation of images from new perspectives, and simplified tasks such as dynamic reconstruction, geometric editing, and physical simulation. Furthermore, the widespread application of consumer-grade RGB-D cameras has promoted the progress of indoor scene reconstruction technology. However, when dealing with objects that are reflective, highly lit, or transparent, as well as distant objects, the depth images produced by these cameras often contain considerable noise and incomplete data, posing challenges to existing systems. This survey comprehensively reviews the latest progress in indoor scene reconstruction using current technologies and consumer-grade cameras, covering aspects off depth image processing, camera pose estimation, surface fusion, optimization, and data completion. By comparing and analyzing the latest methods, this survey aims to provide beginners with an introduction to the field and offer experienced researchers a comprehensive technical overview, thereby fostering the continuous development of this field.

computer science, information systems,telecommunications,engineering, electrical & electronic
Structure Preserving Large Imagery Reconstruction

Ju Shen,Jianjun Yang,Sami Taha-abusneineh,Bryson Payne,Markus Hitz

DOI: https://doi.org/10.48550/arXiv.1409.3906

2014-09-13

Abstract:With the explosive growth of web-based cameras and mobile devices, billions of photographs are uploaded to the internet. We can trivially collect a huge number of photo streams for various goals, such as image clustering, 3D scene reconstruction, and other big data applications. However, such tasks are not easy due to the fact the retrieved photos can have large variations in their view perspectives, resolutions, lighting, noises, and distortions. Fur-thermore, with the occlusion of unexpected objects like people, vehicles, it is even more challenging to find feature correspondences and reconstruct re-alistic scenes. In this paper, we propose a structure-based image completion algorithm for object removal that produces visually plausible content with consistent structure and scene texture. We use an edge matching technique to infer the potential structure of the unknown region. Driven by the estimated structure, texture synthesis is performed automatically along the estimated curves. We evaluate the proposed method on different types of images: from highly structured indoor environment to natural scenes. Our experimental results demonstrate satisfactory performance that can be potentially used for subsequent big data processing, such as image localization, object retrieval, and scene reconstruction. Our experiments show that this approach achieves favorable results that outperform existing state-of-the-art techniques.

Computer Vision and Pattern Recognition
Real-time Acquisition and Reconstruction of Dynamic Volumes with Neural Structured Illumination

Yixin Zeng,Zoubin Bi,Yin Mingrui,Xiang Feng,Kun Zhou,Hongzhi Wu

DOI: https://doi.org/10.1109/cvpr52733.2024.01908

2024-01-01

Abstract:We propose a novel framework for real-time acquisition and reconstruction of temporally-varying 3D phenomena with high quality. The core of our framework is a deep neural network, with an encoder that directly maps to the structured illumination during acquisition, a decoder that predicts a 1D density distribution from single-pixel measurements under the optimized lighting, and an aggregation module that combines the predicted densities for each camera into a single volume. It enables the automatic and joint optimization of physical acquisition and computational reconstruction, and is flexible to adapt to different hardware configurations. The effectiveness of our framework is demonstrated on a lightweight setup with an off-the-shelf projector and one or multiple cameras, achieving a performance of 40 volumes per second at a spatial resolution of 128 3 . We compare favorably with state-of-the-art techniques in real and synthetic experiments, and evaluate the impact of various factors over our pipeline.
Long-LRM: Long-sequence Large Reconstruction Model for Wide-coverage Gaussian Splats

Chen Ziwen,Hao Tan,Kai Zhang,Sai Bi,Fujun Luan,Yicong Hong,Li Fuxin,Zexiang Xu

2024-10-17

Abstract:We propose Long-LRM, a generalizable 3D Gaussian reconstruction model that is capable of reconstructing a large scene from a long sequence of input images. Specifically, our model can process 32 source images at 960x540 resolution within only 1.3 seconds on a single A100 80G GPU. Our architecture features a mixture of the recent Mamba2 blocks and the classical transformer blocks which allowed many more tokens to be processed than prior work, enhanced by efficient token merging and Gaussian pruning steps that balance between quality and efficiency. Unlike previous feed-forward models that are limited to processing 1~4 input images and can only reconstruct a small portion of a large scene, Long-LRM reconstructs the entire scene in a single feed-forward step. On large-scale scene datasets such as DL3DV-140 and Tanks and Temples, our method achieves performance comparable to optimization-based approaches while being two orders of magnitude more efficient. Project page: <a class="link-external link-https" href="https://arthurhero.github.io/projects/llrm" rel="external noopener nofollow">this https URL</a>

Computer Vision and Pattern Recognition
DrivingRecon: Large 4D Gaussian Reconstruction Model For Autonomous Driving

Hao Lu,Tianshuo Xu,Wenzhao Zheng,Yunpeng Zhang,Wei Zhan,Dalong Du,Masayoshi Tomizuka,Kurt Keutzer,Yingcong Chen

2024-12-12

Abstract:Photorealistic 4D reconstruction of street scenes is essential for developing real-world simulators in autonomous driving. However, most existing methods perform this task offline and rely on time-consuming iterative processes, limiting their practical applications. To this end, we introduce the Large 4D Gaussian Reconstruction Model (DrivingRecon), a generalizable driving scene reconstruction model, which directly predicts 4D Gaussian from surround view videos. To better integrate the surround-view images, the Prune and Dilate Block (PD-Block) is proposed to eliminate overlapping Gaussian points between adjacent views and remove redundant background points. To enhance cross-temporal information, dynamic and static decoupling is tailored to better learn geometry and motion features. Experimental results demonstrate that DrivingRecon significantly improves scene reconstruction quality and novel view synthesis compared to existing methods. Furthermore, we explore applications of DrivingRecon in model pre-training, vehicle adaptation, and scene editing. Our code is available at <a class="link-external link-https" href="https://github.com/EnVision-Research/DriveRecon" rel="external noopener nofollow">this https URL</a>.

Computer Vision and Pattern Recognition
PanoRecon: Real-Time Panoptic 3D Reconstruction from Monocular Video

Dong Wu,Zike Yan,Hongbin Zha

DOI: https://doi.org/10.1109/cvpr52733.2024.02032

2024-01-01

Abstract:We introduce the Panoptic 3D Reconstruction task, a unified and holistic scene understanding task for a monocular video. And we present PanoRecon - a novel framework to address this new task, which realizes an online geometry reconstruction alone with dense semantic and instance labeling. Specifically, PanoRecon incrementally performs panoptic 3D reconstruction for each video fragment consisting of multiple consecutive key frames, from a volumetric feature representation using feed-forward neural networks. We adopt a depth-guided back-projection strategy to sparse and purify the volumetric feature representation. We further introduce a voxel clustering module to get object instances in each local fragment, and then design a tracking and fusion algorithm for the integration of instances from different fragments to ensure temporal co-herence. Such design enables our PanoRecon to yield a coherent and accurate panoptic 3D reconstruction. Exper-iments on ScanNetV2 demonstrate a very competitive geometry reconstruction result compared with state-of-the-art reconstruction methods, as well as promising 3D panoptic segmentation result with only RGB input, while being real-time. Code is available at: https://github.com/Riser6/PanoRecon.
Factorized and Controllable Neural Re-Rendering of Outdoor Scene for Photo Extrapolation

Boming Zhao,Bangbang Yang,Zhenyang Li,Zuoyue Li,Guofeng Zhang,Jiashu Zhao,Dawei Yin,Zhaopeng Cui,Hujun Bao

DOI: https://doi.org/10.48550/arXiv.2207.06899

2022-07-14

Computer Vision and Pattern Recognition

Abstract:Expanding an existing tourist photo from a partially captured scene to a full scene is one of the desired experiences for photography applications. Although photo extrapolation has been well studied, it is much more challenging to extrapolate a photo (i.e., selfie) from a narrow field of view to a wider one while maintaining a similar visual style. In this paper, we propose a factorized neural re-rendering model to produce photorealistic novel views from cluttered outdoor Internet photo collections, which enables the applications including controllable scene re-rendering, photo extrapolation and even extrapolated 3D photo generation. Specifically, we first develop a novel factorized re-rendering pipeline to handle the ambiguity in the decomposition of geometry, appearance and illumination. We also propose a composited training strategy to tackle the unexpected occlusion in Internet images. Moreover, to enhance photo-realism when extrapolating tourist photographs, we propose a novel realism augmentation process to complement appearance details, which automatically propagates the texture details from a narrow captured photo to the extrapolated neural rendered image. The experiments and photo editing examples on outdoor scenes demonstrate the superior performance of our proposed method in both photo-realism and downstream applications.
EPRecon: An Efficient Framework for Real-Time Panoptic 3D Reconstruction from Monocular Video

Zhen Zhou,Yunkai Ma,Junfeng Fan,Shaolin Zhang,Fengshui Jing,Min Tan

2024-09-20

Abstract:Panoptic 3D reconstruction from a monocular video is a fundamental perceptual task in robotic scene understanding. However, existing efforts suffer from inefficiency in terms of inference speed and accuracy, limiting their practical applicability. We present EPRecon, an efficient real-time panoptic 3D reconstruction framework. Current volumetric-based reconstruction methods usually utilize multi-view depth map fusion to obtain scene depth priors, which is time-consuming and poses challenges to real-time scene reconstruction. To address this issue, we propose a lightweight module to directly estimate scene depth priors in a 3D volume for reconstruction quality improvement by generating occupancy probabilities of all voxels. In addition, compared with existing panoptic segmentation methods, EPRecon extracts panoptic features from both voxel features and corresponding image features, obtaining more detailed and comprehensive instance-level semantic information and achieving more accurate segmentation results. Experimental results on the ScanNetV2 dataset demonstrate the superiority of EPRecon over current state-of-the-art methods in terms of both panoptic 3D reconstruction quality and real-time inference. Code is available at <a class="link-external link-https" href="https://github.com/zhen6618/EPRecon" rel="external noopener nofollow">this https URL</a>.

Computer Vision and Pattern Recognition
360Recon: An Accurate Reconstruction Method Based on Depth Fusion from 360 Images

Zhongmiao Yan,Qi Wu,Songpengcheng Xia,Junyuan Deng,Xiang Mu,Renbiao Jin,Ling Pei

2024-11-28

Abstract:360-degree images offer a significantly wider field of view compared to traditional pinhole cameras, enabling sparse sampling and dense 3D reconstruction in low-texture environments. This makes them crucial for applications in VR, AR, and related fields. However, the inherent distortion caused by the wide field of view affects feature extraction and matching, leading to geometric consistency issues in subsequent multi-view reconstruction. In this work, we propose 360Recon, an innovative MVS algorithm for ERP images. The proposed spherical feature extraction module effectively mitigates distortion effects, and by combining the constructed 3D cost volume with multi-scale enhanced features from ERP images, our approach achieves high-precision scene reconstruction while preserving local geometric consistency. Experimental results demonstrate that 360Recon achieves state-of-the-art performance and high efficiency in depth estimation and 3D reconstruction on existing public panoramic reconstruction datasets.

Computer Vision and Pattern Recognition
Panoramic 3D Reconstruction by Fusing Color Intensity and Laser Range Data

Wei Jiang,Jian Lu

DOI: https://doi.org/10.1541/ieejeiss.127.568

2007-01-01

Abstract:Technology for capturing panoramic (360 degrees) three-dimensional information in a real environment have many applications in fields: virtual and complex reality, security, robot navigation, and so forth. In this study, we examine an acquisition device constructed of a regular CCD camera and a 2D laser range scanner, along with a technique for panoramic 3D reconstruction using a data fusion algorithm based on an energy minimization framework. The acquisition device can capture two types of data of a panoramic scene without occlusion between two sensors: a dense spatio-temporal volume from a camera and distance information from a laser scanner. We resample the dense spatio-temporal volume for generating a dense multi-perspective panorama that has equal spatial resolution to that of the original images acquired using a regular camera, and also estimate a dense panoramic depth-map corresponding to the generated reference panorama by extracting trajectories from the dense spatio-temporal volume with a selecting camera. Moreover, for determining distance information robustly, we propose a data fusion algorithm that is embedded into an energy minimization framework that incorporates active depth measurements using a 2D laser range scanner and passive geometry reconstruction from an image sequence obtained using the CCD camera. Thereby, measurement precision and robustness can be improved beyond those available by conventional methods using either passive geometry reconstruction (stereo vision) or a laser range scanner. Experimental results using both synthetic and actual images show that our approach can produce high-quality panoramas and perform accurate 3D reconstruction in a panoramic environment.

Toward Large-Scale Plenoptic Reconstruction

Plenoptic Reconstruction

Theory and Approach of Large-Scale Computational Reconstruction

Large-Scale 3D Reconstruction from Multi-View Imagery: A Comprehensive Review

DP-MVS: Detail Preserving Multi-View Surface Reconstruction of Large-Scale Scenes

Outdoor Scene 3D Reconstruction from Multiple Point Cloud

GigaVision: when Computer Vision Meets Gigapixel Videography

A Divide-and-conquer Approach to Large Scene Reconstruction with Interactive Scene Analysis and Segmentation

VastGaussian: Vast 3D Gaussians for Large Scene Reconstruction

High-Resolution Plenoptic Sensing

A New Era of Indoor Scene Reconstruction: A Survey

Structure Preserving Large Imagery Reconstruction

Real-time Acquisition and Reconstruction of Dynamic Volumes with Neural Structured Illumination

Long-LRM: Long-sequence Large Reconstruction Model for Wide-coverage Gaussian Splats

DrivingRecon: Large 4D Gaussian Reconstruction Model For Autonomous Driving

PanoRecon: Real-Time Panoptic 3D Reconstruction from Monocular Video

Factorized and Controllable Neural Re-Rendering of Outdoor Scene for Photo Extrapolation

EPRecon: An Efficient Framework for Real-Time Panoptic 3D Reconstruction from Monocular Video

360Recon: An Accurate Reconstruction Method Based on Depth Fusion from 360 Images

Panoramic 3D Reconstruction by Fusing Color Intensity and Laser Range Data