Ghost on the Shell: An Expressive Representation of General 3D Shapes

Zhen Liu,Yao Feng,Yuliang Xiu,Weiyang Liu,Liam Paull,Michael J. Black,Bernhard Schölkopf

2024-03-25

Abstract:The creation of photorealistic virtual worlds requires the accurate modeling of 3D surface geometry for a wide range of objects. For this, meshes are appealing since they 1) enable fast physics-based rendering with realistic material and lighting, 2) support physical simulation, and 3) are memory-efficient for modern graphics pipelines. Recent work on reconstructing and statistically modeling 3D shape, however, has critiqued meshes as being topologically inflexible. To capture a wide range of object shapes, any 3D representation must be able to model solid, watertight, shapes as well as thin, open, surfaces. Recent work has focused on the former, and methods for reconstructing open surfaces do not support fast reconstruction with material and lighting or unconditional generative modelling. Inspired by the observation that open surfaces can be seen as islands floating on watertight surfaces, we parameterize open surfaces by defining a manifold signed distance field on watertight templates. With this parameterization, we further develop a grid-based and differentiable representation that parameterizes both watertight and non-watertight meshes of arbitrary topology. Our new representation, called Ghost-on-the-Shell (G-Shell), enables two important applications: differentiable rasterization-based reconstruction from multiview images and generative modelling of non-watertight meshes. We empirically demonstrate that G-Shell achieves state-of-the-art performance on non-watertight mesh reconstruction and generation tasks, while also performing effectively for watertight meshes.

Graphics,Machine Learning

What problem does this paper attempt to address?

The problem that this paper attempts to solve is **how to represent and generate general 3D shapes, especially non - watertight open surfaces, efficiently**. Specifically, existing 3D shape reconstruction and generation methods have some limitations when dealing with non - watertight surfaces, such as:

1. **Modeling of non - watertight surfaces**: Many existing methods mainly focus on the reconstruction of closed surfaces and lack sufficient support for non - watertight surfaces. Non - watertight surfaces such as clothes, paper or leaves, etc. The surfaces of these objects are non - closed and open, and their capture and generative modeling have been relatively less explored.

2. **Reconstruction efficiency and quality**: For the reconstruction of non - watertight surfaces, existing methods usually require complex post - processing steps, which not only increase the computational cost but may also introduce additional errors. In addition, these methods often do not support rapid material and illumination reconstruction and are not suitable for unconditional generative modeling.

3. **Topological flexibility**: Existing 3D representation methods have limitations when dealing with shapes of different topological structures, especially when dealing with non - watertight surfaces, it is difficult to flexibly capture various topological changes.

To overcome these problems, the paper proposes a new representation method - **Ghost - on - the - Shell (G - S HELL)**, which can effectively parameterize closed and non - closed 3D meshes and support efficient reconstruction and generation tasks. The main contributions of G - S HELL include:

- **Mesh representation**: G - S HELL is a differentiable representation method that can effectively parameterize closed and non - closed meshes of different topological structures.
- **Efficiency**: By designing an efficient mesh extraction algorithm, G - S HELL can achieve rapid reconstruction of non - closed meshes and support differentiable rasterization.
- **Physics - based inverse rendering**: G - S HELL allows simultaneous optimization of the topology, material and illumination of closed and non - closed meshes.
- **Mesh generation**: The mesh parameterization and efficient mesh extraction of G - S HELL enable the diffusion model to be applied to the generation of non - closed meshes for the first time.

In summary, this paper aims to provide a new 3D shape representation method to solve the limitations of existing methods in dealing with non - watertight surfaces and improve the efficiency and quality of reconstruction and generation tasks.