Abstract:Recently, learning-based approaches for 3D model reconstruction have attracted attention owing to its modern applications such as Extended Reality(XR), robotics and self-driving cars. Several approaches presented good performance on reconstructing 3D shapes by learning solely from images, i.e., without using 3D models in training. Challenges, however, remain in texture generation due to the gap between 2D and 3D modals. In previous work, the grid sampling mechanism from Spatial Transformer Networks was adopted to sample color from an input image to formulate texture. Despite its success, the existing framework has limitations on searching scope in sampling, resulting in flaws in generated texture and consequentially on rendered 3D models. In this paper, to solve that issue, we present a novel sampling algorithm by optimizing the gradient of predicted coordinates based on the variance on the sampling image. Taking into account the semantics of the image, we adopt Frechet Inception Distance (FID) to form a loss function in learning, which helps bridging the gap between rendered images and input images. As a result, we greatly improve generated texture. Furthermore, to optimize 3D shape reconstruction and to accelerate convergence at training, we adopt part segmentation and template learning in our model. Without any 3D supervision in learning, and with only a collection of single-view 2D images, the shape and texture learned by our model outperform those from previous work. We demonstrate the performance with experimental results on a publically available dataset.

Comparison of Two-Dimensional Integration Methods for Shape Reconstruction from Gradient Data

Integration Algorithm in Three-dimensional Plane Shape Measuring by Fringe Reflection

Shape reconstruction from gradient data in an arbitrarily-shaped aperture by iterative discrete cosine transforms in Southwell configuration

Shape from Gradient Fields Using Kernel Poisson Equation

Framework for gradient integration by combining radial basis functions method and least-squares method.

Improvement of Least-Squares Integration Method with Iterative Compensations in Fringe Reflectometry

Convergence Analysis of Gradient Algorithms on Riemannian Manifolds Without Curvature Constraints and Application to Riemannian Mass

Adaptive Sampling and Reconstruction for Gradient-Domain Rendering

A unification of least-squares and Green-Gauss gradients under a common projection-based gradient reconstruction framework

Shape Reconstruction for Unsteady Advection-Diffusion Problems by Domain Derivative Method

A geometric integration approach to smooth optimisation: Foundations of the discrete gradient method

Similarity and Comparison of Three Finite-Volume Methods for Diffusive Fluxes Computation on Nonorthogonal Meshes

A generalized formulation for gradient schemes in unstructured finite volume method

Spline Based Least Squares Integration for Two-Dimensional Shape or Wavefront Reconstruction

Quantitative study of 3D gradient operators

Shape Reconstruction of an Inverse Boundary Value Problem of Two‐dimensional Navier–Stokes Equations

CALCULATIONS OF TWO-DIMENSIONAL TRANSIENT POTENTIAL AND ITS GRADIENT BY COMBINING DIRECT METHOD AND HILBERT TRANSFORM

An Integrated Linear Reconstruction for Finite Volume Scheme on Unstructured Grids

A Shape From Shading Algorithm And Its Application

Using Adaptive Gradient for Texture Learning in Single-View 3D Reconstruction

ON AUTOMATIC DIFFERENTIATION AND ALGORITHMIC LINEARIZATION