Abstract:N:M sparsity has received increasing attention due to its remarkable performance and latency trade-off compared with structured and unstructured sparsity. However, existing N:M sparsity methods do not differentiate the relative importance of weights among blocks and leave important weights underappreciated. Besides, they directly apply N:M sparsity to the whole network, which will cause severe information loss. Thus, they are still sub-optimal. In this paper, we propose an efficient and effective Multi-Axis Query methodology, dubbed as MaxQ, to rectify these problems. During the training, MaxQ employs a dynamic approach to generate soft N:M masks, considering the weight importance across multiple axes. This method enhances the weights with more importance and ensures more effective updates. Meanwhile, a sparsity strategy that gradually increases the percentage of N:M weight blocks is applied, which allows the network to heal from the pruning-induced damage progressively. During the runtime, the N:M soft masks can be precomputed as constants and folded into weights without causing any distortion to the sparse pattern and incurring additional computational overhead. Comprehensive experiments demonstrate that MaxQ achieves consistent improvements across diverse CNN architectures in various computer vision tasks, including image classification, object detection and instance segmentation. For ResNet50 with 1:16 sparse pattern, MaxQ can achieve 74.6% top-1 accuracy on ImageNet and improve by over 2.8% over the state-of-the-art.

Efficient Approximate Maximum Inner Product Search over Sparse Vectors

Clustering is Efficient for Approximate Maximum Inner Product Search

FARGO: Fast Maximum Inner Product Search Via Global Multi-Probing.

Query-Aware Quantization for Maximum Inner Product Search.

ProMIPS: Efficient High-Dimensional C-Approximate Maximum Inner Product Search with a Lightweight Index

A Greedy Approach for Budgeted Maximum Inner Product Search

Anisotropic Additive Quantization for Fast Inner Product Search

MaxQ: Multi-Axis Query for N:M Sparsity Network

ISIS: a new approach for efficient similarity search in sparse databases

Learning Binary Codes For Maximum Inner Product Search

Approximate Vector Set Search: A Bio-Inspired Approach for High-Dimensional Spaces

Efficient and Effective Retrieval of Dense-Sparse Hybrid Vectors using Graph-based Approximate Nearest Neighbor Search

Optimistic Query Routing in Clustering-based Approximate Maximum Inner Product Search

Orthogonal Projective Sparse Coding for Image Representation

Exact and Approximate Maximum Inner Product Search with LEMP

I/O Efficient Approximate Nearest Neighbour Search Based on Learned Functions.

On the Complexity of Inner Product Similarity Join

A New Spherical Hashing Method in a Low-Dimensional Isotropic Space.

SIMD-Optimized Search Over Sorted Data

A Two-Level Simultaneous Orthogonal Matching Pursuit Algorithm for Simultaneous Sparse Approximation Problems

Kernel Based Latent Semantic Sparse Hashing for Large-Scale Retrieval from Heterogeneous Data Sources