Abstract:Deep neural networks are widely used for understanding 3D point clouds. At each point convolution layer, features are computed from local neighborhoods of 3D points and combined for subsequent processing in order to extract semantic information. Existing methods adopt the same individual point neighborhoods throughout the network layers, defined by the same metric on the fixed input point coordinates. This common practice is easy to implement but not necessarily optimal. Ideally, local neighborhoods should be different at different layers, as more latent information is extracted at deeper layers. We propose a novel end-to-end approach to learn different non-rigid transformations of the input point cloud so that optimal local neighborhoods can be adopted at each layer. We propose both linear (affine) and non-linear (projective and deformable) spatial transformers for 3D point clouds. With spatial transformers on the ShapeNet part segmentation dataset, the network achieves higher accuracy for all categories, with 8\% gain on earphones and rockets in particular. Our method also outperforms the state-of-the-art on other point cloud tasks such as classification, detection, and semantic segmentation. Visualizations show that spatial transformers can learn features more efficiently by dynamically altering local neighborhoods according to the geometry and semantics of 3D shapes in spite of their within-category variations. Our code is publicly available at <a class="link-external link-https" href="https://github.com/samaonline/spatial-transformer-for-3d-point-clouds" rel="external noopener nofollow">this https URL</a>.

P2MAT-NET: Learning Medial Axis Transform from Sparse Point Clouds

MAT-Net: Medial Axis Transform Network for 3D Object Recognition

IMMAT: Mesh Reconstruction from Single View Images by Medial Axis Transform Prediction

SEG-MAT: 3D Shape Segmentation Using Medial Axis Transform

A survey on geometric shape representation of objects based on medial axis transform

AMAT: Medial Axis Transform for Natural Images

Computing Medial Axis Transform with Feature Preservation via Restricted Power Diagram

Computing Medial Axis Transformations of 2D Point Clouds.

Q-MAT: Computing Medial Axis Transform By Quadratic Error Minimization.

Spatial Transformer for 3D Point Clouds

A Unified Feature Representation and Learning Framework for 3D Shape

Shape registration with learned deformations for 3D shape reconstruction from sparse and incomplete point clouds

2D3D-MATR: 2D-3D Matching Transformer for Detection-free Registration between Images and Point Clouds

A Metric Learning Method for Image-based 3D Shape Retrieval

3D-Isrnet

Q-MAT+: An error-controllable and feature-sensitive simplification algorithm for medial axis transform

PointMTL: Multi-Transform Learning for Effective 3D Point Cloud Representations

Quasi-Medial Distance Field (Q-MDF): A Robust Method for Approximating and Discretizing Neural Medial Axis

PCPNET: Learning Local Shape Properties from Raw Point Clouds

Triangle-Net: Towards Robustness in Point Cloud Learning

T-Pixel2Mesh: Combining Global and Local Transformer for 3D Mesh Generation from a Single Image