Abstract:N:M sparsity has received increasing attention due to its remarkable performance and latency trade-off compared with structured and unstructured sparsity. However, existing N:M sparsity methods do not differentiate the relative importance of weights among blocks and leave important weights underappreciated. Besides, they directly apply N:M sparsity to the whole network, which will cause severe information loss. Thus, they are still sub-optimal. In this paper, we propose an efficient and effective Multi-Axis Query methodology, dubbed as MaxQ, to rectify these problems. During the training, MaxQ employs a dynamic approach to generate soft N:M masks, considering the weight importance across multiple axes. This method enhances the weights with more importance and ensures more effective updates. Meanwhile, a sparsity strategy that gradually increases the percentage of N:M weight blocks is applied, which allows the network to heal from the pruning-induced damage progressively. During the runtime, the N:M soft masks can be precomputed as constants and folded into weights without causing any distortion to the sparse pattern and incurring additional computational overhead. Comprehensive experiments demonstrate that MaxQ achieves consistent improvements across diverse CNN architectures in various computer vision tasks, including image classification, object detection and instance segmentation. For ResNet50 with 1:16 sparse pattern, MaxQ can achieve 74.6% top-1 accuracy on ImageNet and improve by over 2.8% over the state-of-the-art.

Typical sample selection and redundancy reduction for min-max modular network with GZC function

An algorithm for pruning redundant modules in min-max modular network with GZC function

A Min-Max Modular Network with Gaussian-Zero-Crossing Function

A Gaussian Zero-Crossing Discriminant Function For Min-Max Modular Neural Networks

A Neural Network Model For General Minimax Problem

Be expert in multiple aspects and good at many modular neural network with reduced subnet relative training complexity

A Gaussian Zero Crossing Discriminant Function

An Algorithm for Pruning Redundant Modules in Min-Ma Modular Network

MaxQ: Multi-Axis Query for N:M Sparsity Network

ModularBoost: an Efficient Network Inference Algorithm Based on Module Decomposition.

Emergent online learning with a Gaussian zero-crossing discriminant function

A New Supervised Clustering Algorithm Based on Min-Max Modular Network with Gaussian-Zero-Crossing Functions

Module Combination Based on Decision Tree in Min-Max Modular Network

Fault Diagnosis for Industrial Images Using a Min-Max Modular Neural Network

Structure pruning strategies for min-max modular network

On Efficient Selection of Binary Classifiers for Min-Max Modular Classifier

Parallelized extreme learning machine ensemble based on min-max modular network

Improvement On Response Performance Of Min-Max Modular Classifier By Symmetric Module Selection

Emergent on-line learning in min-max modular neural networks

Learning Multiple Criteria Calibration for Generalized Zero-shot Learning