Abstract:Structured pruning is one of the most popular approaches to effectively compress the heavy deep neural networks (DNNs) into compact sub-networks while retaining performance. The existing methods suffer from multi-stage procedures along with significant engineering efforts and human expertise. The Only-Train-Once (OTO) series has been recently proposed to resolve the many pain points by streamlining the workflow by automatically conducting (i) search space generation, (ii) structured sparse optimization, and (iii) sub-network construction. However, the built-in sparse optimizers in the OTO series, i.e., the Half-Space Projected Gradient (HSPG) family, have limitations that require hyper-parameter tuning and the implicit controls of the sparsity exploration, consequently requires intervening by human expertise. To address such limitations, we propose a Hybrid Efficient Structured Sparse Optimizer (HESSO). HESSO could automatically and efficiently train a DNN to produce a high-performing subnetwork. Meanwhile, it is almost tuning-free and enjoys user-friendly integration for generic training applications. To address another common issue of irreversible performance collapse observed in pruning DNNs, we further propose a Corrective Redundant Identification Cycle (CRIC) for reliably identifying indispensable structures. We numerically demonstrate the efficacy of HESSO and its enhanced version HESSO-CRIC on a variety of applications ranging from computer vision to natural language processing, including large language model. The numerical results showcase that HESSO can achieve competitive even superior performance to varying state-of-the-arts and support most DNN architectures. Meanwhile, CRIC can effectively prevent the irreversible performance collapse and further enhance the performance of HESSO on certain applications. The code is available at <a class="link-external link-https" href="https://github.com/microsoft/only_train_once" rel="external noopener nofollow">this https URL</a>.

ODEN: Live Programming for Neural Network Architecture Editing

An In-Situ Visual Analytics Framework for Deep Neural Networks

SpaceEditing: Integrating Human Knowledge into Deep Neural Networks via Interactive Latent Space Editing

An optimised deep spiking neural network architecture without gradients

Towards automated neural design: An open source, distributed neural architecture research framework

OTOv3: Automatic Architecture-Agnostic Neural Network Training and Compression from Structured Pruning to Erasing Operators

Monitizer: Automating Design and Evaluation of Neural Network Monitors

Using Neural Networks for Programming by Demonstration

On Tuning Neural ODE for Stability, Consistency and Faster Convergence

A Visual Programming Paradigm for Abstract Deep Learning Model Development

ODEN: A Framework to Solve Ordinary Differential Equations using Artificial Neural Networks

Enabling Computer Vision Driven Assistive Devices for the Visually Impaired via Micro-architecture Design Exploration

Isolation-Based Debugging for Neural Networks

KAN-ODEs: Kolmogorov-Arnold Network Ordinary Differential Equations for Learning Dynamical Systems and Hidden Physics

Exploring the Evolution of Hidden Activations with Live-Update Visualization

Quantum Neural Network Software Testing, Analysis, and Code Optimization for Advanced IoT Systems: Design, Implementation, and Visualization

Skyline: Interactive In-Editor Computational Performance Profiling for Deep Neural Network Training

Revealing hidden dynamics from time-series data by ODENet

Design of Explainability Module with Experts in the Loop for Visualization and Dynamic Adjustment of Continual Learning

NeuralVis: Visualizing and Interpreting Deep Learning Models

HESSO: Towards Automatic Efficient and User Friendly Any Neural Network Training and Pruning