Abstract:The great success of deep neural networks is built upon their over-parameterization, which smooths the optimization landscape without degrading the generalization ability. Despite the benefits of over-parameterization, a huge amount of parameters makes deep networks cumbersome in daily life applications. On the other hand, training neural networks without over-parameterization faces many practical problems, e.g., being trapped in the local optimal. Though techniques such as pruning and distillation are developed, they are expensive in fully training a dense network as backward selection methods; and there is still a void on systematically exploring forward selection methods for learning structural sparsity in deep networks. To fill in this gap, this paper proposes a new approach based on differential inclusions of inverse scale spaces. Specifically, our method can generate a family of models from simple to complex ones along the dynamics via coupling a pair of parameters, such that over-parameterized deep models and their structural sparsity can be explored simultaneously. This kind of differential inclusion scheme has a simple discretization, dubbed Deep structure splitting Linearized Bregman Iteration (DessiLBI), whose global convergence in learning deep networks could be established under the Kurdyka-Łojasiewicz framework. Particularly, we explore several applications of DessiLBI, including finding sparse structures of networks directly via the coupled structure parameter and growing networks from simple to complex ones progressively. Experimental evidence shows that our method achieves comparable and even better performance than the competitive optimizers in exploring the sparse structure of several widely used backbones on the benchmark datasets. Remarkably, with early stopping, our method unveils "winning tickets" in early epochs: the effective sparse network structures with comparable test accuracy to fully trained over-parameterized models, that are further transferable to similar alternative tasks. Furthermore, our method is able to grow networks efficiently with adaptive filter configurations, demonstrating the good performance with much less computational cost. Codes and models can be downloaded at https://github.com/DessiLBI2020/DessiLBI.

Dynamic Connectivity of Delay Loops in Folded-in-Time Dnn

Single-layer Folded RNN for Time Series Prediction and Classification under a Non-Von Neumann Architecture

Condense: A Framework for Device and Frequency Adaptive Neural Network Models on the Edge.

Efficient Structure Slimming for Spiking Neural Networks

Dynamic Sparse Graph for Efficient Deep Learning.

Dynamical Optimal Learning For Fnn And Its Applications

Biologically Inspired Structure Learning with Reverse Knowledge Distillation for Spiking Neural Networks

Deep CovDenseSNN: A Hierarchical Event-Driven Dynamic Framework with Spiking Neurons in Noisy Environment

Toward a Brain-Inspired Developmental Neural Network Based on Dendritic Spine Dynamics

Exploring Structural Sparsity of Deep Networks Via Inverse Scale Spaces

Improving deep neural networks for LVCSR using dropout and shrinking structure

Dynamic Deep Neural Networks: Optimizing Accuracy-Efficiency Trade-Offs by Selective Execution

A Discrete Version of the Dynamic Link Network.

Dynamic Neural Dowker Network: Approximating Persistent Homology in Dynamic Directed Graphs

Delay learning based on temporal coding in Spiking Neural Networks

On the Application and Compression of Deep Time Delay Neural Network for Embedded Statistical Parametric Speech Synthesis

Efficient Synthesis of Compact Deep Neural Networks

DTS: Dynamic Training Slimming with Feature Sparsity for Efficient Convolutional Neural Network

E-LSTM-D: A Deep Learning Framework for Dynamic Network Link Prediction

Deep Time Delay Neural Network for Speech Enhancement with Full Data Learning

LDP: Learnable Dynamic Precision for Efficient Deep Neural Network Training and Inference