OA-CNNs: Omni-Adaptive Sparse CNNs for 3D Semantic Segmentation

Bohao Peng,Xiaoyang Wu,Li Jiang,Yukang Chen,Hengshuang Zhao,Zhuotao Tian,Jiaya Jia

2024-03-21

Abstract:The booming of 3D recognition in the 2020s began with the introduction of point cloud transformers. They quickly overwhelmed sparse CNNs and became state-of-the-art models, especially in 3D semantic segmentation. However, sparse CNNs are still valuable networks, due to their efficiency treasure, and ease of application. In this work, we reexamine the design distinctions and test the limits of what a sparse CNN can achieve. We discover that the key credit to the performance difference is adaptivity. Specifically, we propose two key components, i.e., adaptive receptive fields (spatially) and adaptive relation, to bridge the gap. This exploration led to the creation of Omni-Adaptive 3D CNNs (OA-CNNs), a family of networks that integrates a lightweight module to greatly enhance the adaptivity of sparse CNNs at minimal computational cost. Without any self-attention modules, OA-CNNs favorably surpass point transformers in terms of accuracy in both indoor and outdoor scenes, with much less latency and memory cost. Notably, it achieves 76.1%, 78.9%, and 70.6% mIoU on ScanNet v2, nuScenes, and SemanticKITTI validation benchmarks respectively, while maintaining at most 5x better speed than transformer counterparts. This revelation highlights the potential of pure sparse CNNs to outperform transformer-related networks.

Computer Vision and Pattern Recognition

What problem does this paper attempt to address?

### Problems the Paper Aims to Solve

The paper aims to address the performance gap between Sparse Convolutional Neural Networks (Sparse CNNs) and Point Transformers in 3D semantic segmentation. Specifically, while Sparse CNNs have advantages in efficiency and ease of application, their accuracy is generally inferior to transformer-based models, especially in 3D scene semantic segmentation tasks.

By re-examining the design differences of Sparse CNNs and testing their limits, the authors found that adaptivity is the key factor in the performance gap. To this end, they propose two key components:

1. **Adaptive Receptive Fields**: Spatially adaptive receptive fields that can adjust their size according to the characteristics of different objects and regions.
2. **Adaptive Relation**: Enhancing the adaptivity of Sparse CNNs by introducing adaptive relation mapping.

With these improvements, the authors developed Omni-Adaptive 3D CNNs (OA-CNNs), a family of networks that integrate lightweight modules to significantly enhance the adaptivity of Sparse CNNs with minimal computational cost. Experimental results show that OA-CNNs not only surpass Point Transformer models in accuracy for both indoor and outdoor scenes but also have significant advantages in latency and memory consumption. For example, in the ScanNet v2, nuScenes, and SemanticKITTI benchmarks, OA-CNNs achieved mIoU scores of 76.1%, 78.9%, and 70.6%, respectively, while maintaining a speed 5 times faster than transformer models. This indicates that pure Sparse CNNs have the potential to outperform transformer-based networks.