Abstract:It is common experience for human vision to perceive full 3D shape and scene from a single 2D image with the occluded parts "filled-in" by prior visual knowledge. Thus, computing the 3D structures of all the objects in the scene from a single image is a fundamental problem in computer vision. In this thesis, we propose a bottom-up/top-down Bayesian inference framework to compute the 3D structures of objects in the scene from a single image, which integrates the involved visual tasks (segmentation, perceptual grouping, object detection and recognition, 3D reconstruction) in a principled way and incorporates the prior visual knowledge in the inference. The output of the inference framework is a hierarchical "parsing graph" with the scene label at the top (or root), objects with 3D structures and their parts at intermediate nodes, and image pixels at the bottom. The number of layers in this parsing graph is determined by the types of objects or visual patterns. The nodes in this parsing graph correspond to visual patterns represented by probabilistic models. The parsing graph also has both top-down connections and horizontal spatial connections, which correspond to the generative models and spatial relations modeled by Markov Random Field (MRF) respectively. Formulated in Bayesian framework, the inference algorithm computes the parsing graph from the input image by optimizing a posterior probability. In this optimization process, we integrate two popular computing paradigms in computer vision: generative methods, and discriminative methods. The former formulates the posterior probability to maximize in terms of generative models for images defined by likelihood functions and priors. The latter computes discriminative proposals using some bottom-up tests to drive the maximizing process in the solution space. Thus, the inference algorithm achieves both speed and consistency. We also investigate three mechanisms to efficiently construct the parsing graph based on the properties of visual patterns being computed: bottom-up construction mechanism, top-down construction mechanism, and bottom-up/top-down construction mechanism.

2D image reasoning and 3D interpretation based on perceptual organization

Qualitative Spatial Representation and Reasoning in 3D Space

Reasoning3D -- Grounding and Reasoning in 3D: Fine-Grained Zero-Shot Open-Vocabulary 3D Reasoning Part Segmentation via Large Vision-Language Models

3D Concept Learning and Reasoning from Multi-View Images

Embedding Visual Cognition in 3D Reconstruction from Multi-View Engineering Drawings

Computing three-dimensional scene from a single image by bottom-up/top-down bayesian inference

Reasoning Geometric Commonsense for Single-view 3D Scene Parsing

Situational Awareness Matters in 3D Vision Language Reasoning

3d Stereoscopic Image Generation From Monocular View In Structured Environments

Take A Step Back: Rethinking the Two Stages in Visual Reasoning

Visual Odometry Based 3D-Reconstruction

Inferring 3D Occupancy Fields Through Implicit Reasoning on Silhouette Images

Beyond Point Clouds: Scene Understanding by Reasoning Geometry and Physics

In-Place Panoptic Radiance Field Segmentation with Perceptual Prior for 3D Scene Understanding

Scene Understanding by Reasoning Stability and Safety

Three-Dimensional Structure Measurement And Optimization Method Of Indoor Scene Based On Single Image

Single-Image 3D Scene Parsing Using Geometric Commonsense

Indoor Scene Recognition in 3D

On Completeness of Reasoning about Planar Spatial Relationships in Pictorial Retrieval Systems.

A novel visual organization based on topological perception

Image to Icosahedral Projection for $\mathrm{SO}(3)$ Object Reasoning from Single-View Images