Abstract:Abstract Traditional convolutional neural network (CNN) methods rely on dense tensors, which makes them suboptimal for spatially sparse data. In this paper, we propose a CNN model based on sparse tensors for efficient processing of high-resolution shapes represented as binary voxel occupancy grids. In contrast to a dense CNN that takes the entire voxel grid as input, a sparse CNN processes only on the non-empty voxels, thus reducing the memory and computation overhead caused by the sparse input data. We evaluate our method on two clinically relevant skull reconstruction tasks: (1) given a defective skull, reconstruct the complete skull (i.e., skull shape completion), and (2) given a coarse skull, reconstruct a high-resolution skull with fine geometric details (shape super-resolution). Our method outperforms its dense CNN-based counterparts in the skull reconstruction task quantitatively and qualitatively, while requiring substantially less memory for training and inference. We observed that, on the 3D skull data, the overall memory consumption of the sparse CNN grows approximately linearly during inference with respect to the image resolutions. During training, the memory usage remains clearly below increases in image resolution—an $$\times 8$$ × 8 increase in voxel number leads to less than $$\times 4$$ × 4 increase in memory requirements. Our study demonstrates the effectiveness of using a sparse CNN for skull reconstruction tasks, and our findings can be applied to other spatially sparse problems. We prove this by additional experimental results on other sparse medical datasets, like the aorta and the heart. Project page at https://github.com/Jianningli/SparseCNN .

Sparsity Invariant CNNs

Least Square Estimation Network for Depth Completion

A Sparsity-Invariant Model Via Unifying Depth Prediction and Completion

SparseDC: Depth Completion from sparse and non-uniform inputs

Depth Completion from Sparse LiDAR Data with Depth-Normal Constraints

HMS-Net: Hierarchical Multi-scale Sparsity-invariant Network for Sparse Depth Completion

Sparse Convolutional Neural Networks for Medical Image Analysis

Self-supervised Sparse-to-Dense: Self-supervised Depth Completion from LiDAR and Monocular Camera

Sparse convolutional neural network for high-resolution skull shape completion and shape super-resolution

Depth-Independent Depth Completion via Least Square Estimation

SparseFormer: Attention-based Depth Completion Network

Sparse Auxiliary Networks for Unified Monocular Depth Prediction and Completion

Agspn: Efficient Attention-Gated Spatial Propagation Network for Depth Completion

SparseNet: A Sparse DenseNet for Image Classification

Adaptive Pixel-wise Structured Sparse Network for Efficient CNNs

Depth Edge Guided CNNs for Sparse Depth Upsampling

DeltaCNN: End-to-End CNN Inference of Sparse Frame Differences in Videos

Depth Completion Using a View-constrained Deep Prior

SparseTrain: Exploiting Dataflow Sparsity for Efficient Convolutional Neural Networks Training

Sparse Activity and Sparse Connectivity in Supervised Learning

Efficient Neural Networks with Spatial Wise Sparsity Using Unified Importance Map.