Abstract:Recovering deformable surfaces is an interesting and beneficial research problem for computer vision and image analysis. An effective deformable surface recovery technique can be applied in a variety of applications for surface reconstruction, digital entertainment, medical imaging and Augmented Reality. While considerable research efforts have been devoted to deformable surface modeling and fitting, there are only few schemes available to tackle the deformable surface recovery problem efficiently. This thesis proposes a set of methods to effectively solve the 2D nonrigid shape recovery and 3D deformable surface tracking based on a robust progressive optimization scheme. The presented techniques are also applied to a variety of real-world applications. To tackle the 2D nonrigid shape recovery problem, this thesis first presents a novel progressive finite Newton optimization scheme, which is based on the local feature correspondences. The key of this approach is to formulate the nonrigid shape recovery as an unconstrained quadratic optimization problem which has a closed-form solution for a given set of observations. For the appearance-based method, a deformable Lucas-Kanade algorithm is proposed which triangulates the template image into small patches and constrains the deformation through the second order derivatives of the mesh vertices. It is formulated into a sparse regularized least squares problem which is able to reduce the computational cost and the memory requirement. The inverse compositional algorithm is applied to efficiently solve the optimization problem. Furthermore, we present a fusion approach to take advantage of both the appearance information and the local features. As for the 3D deformable surface recovery, the key challenge arises from the difficulty in estimating a large number of 3D shape parameters from noisy observations. In this thesis, 3D deformable surface tracking is formulated into an unconstrained quadratic problem that can be solved very efficiently by resolving a set of sparse linear equations. Furthermore, the robust progressive finite Newton method developed for nonrigid surface detection is employed to handle the large outliers. Without resorting to an explicit deformable mesh model, the nonrigid surface detection can be treated as a generic regression problem. A novel velocity coherence constraint is imposed on the deformable shape model to regularize the ill-posed optimization problem. To handle the large outliers, a progressive optimization scheme is employed. In addition to the methodologies studied and evaluated in computer vision, this thesis also investigates the nonrigid surface recovery in some real-world multimedia applications, such as Near-duplicate image retrieval and detection. In contrast to conventional approaches, the presented technique can recover an explicit mapping between two near-duplicate images with a few deformation parameters and find out the correct correspondences from noisy data effectively. To make the proposed technique applicable to large-scale applications, an effective multilevel ranking scheme is presented that filters out the irrelevant results in a coarse-to-fine manner. To overcome the extremely small training size challenge, a semi-supervised learning method is employed to improve the performance using unlabeled data. Extensive evaluations show that the presented method is clearly effective than conventional approaches.

DeFormer: Integrating Transformers with Deformable Models for 3D Shape Abstraction from a Single Image

3Deformer: A Common Framework for Image-Guided Mesh Deformation

DeformNet: Free-Form Deformation Network for 3D Shape Reconstruction from a Single Image

Shape-Space Deformer: Unified Visuo-Tactile Representations for Robotic Manipulation of Deformable Objects

3D-TRANS: 3D Hierarchical Transformer for Shape Correspondence Learning.

ShapeFormer: Transformer-based Shape Completion Via Sparse Representation

Deformer: Dynamic Fusion Transformer for Robust Hand Pose Estimation

A Deformation Model to Reduce the Effect of Expressions in 3D Face Recognition

Disentangling Deep Network for Reconstructing 3D Object Shapes from Single 2D Images

DPF-Net: Combining Explicit Shape Priors in Deformable Primitive Field for Unsupervised Structural Reconstruction of 3D Objects

Deformable Surface Recovery and Its Applications

DeformerNet: Learning Bimanual Manipulation of 3D Deformable Objects

Deformable DETR: Deformable Transformers for End-to-End Object Detection

DeformerNet: A Deep Learning Approach to 3D Deformable Object Manipulation

Deformable 3D Fusion: from Partial Dynamic 3D Observations to Complete 4D Models

DTF-Net: Category-Level Pose Estimation and Shape Reconstruction via Deformable Template Field

DeTurb: Atmospheric Turbulence Mitigation with Deformable 3D Convolutions and 3D Swin Transformers

AutoFormer: Searching Transformers for Visual Recognition

DiT-3D: Exploring Plain Diffusion Transformers for 3D Shape Generation

Self-supervised Learning of Implicit Shape Representation with Dense Correspondence for Deformable Objects