Abstract:The great success of deep neural networks is built upon their over-parameterization, which smooths the optimization landscape without degrading the generalization ability. Despite the benefits of over-parameterization, a huge amount of parameters makes deep networks cumbersome in daily life applications. On the other hand, training neural networks without over-parameterization faces many practical problems, e.g., being trapped in the local optimal. Though techniques such as pruning and distillation are developed, they are expensive in fully training a dense network as backward selection methods; and there is still a void on systematically exploring forward selection methods for learning structural sparsity in deep networks. To fill in this gap, this paper proposes a new approach based on differential inclusions of inverse scale spaces. Specifically, our method can generate a family of models from simple to complex ones along the dynamics via coupling a pair of parameters, such that over-parameterized deep models and their structural sparsity can be explored simultaneously. This kind of differential inclusion scheme has a simple discretization, dubbed Deep structure splitting Linearized Bregman Iteration (DessiLBI), whose global convergence in learning deep networks could be established under the Kurdyka-Łojasiewicz framework. Particularly, we explore several applications of DessiLBI, including finding sparse structures of networks directly via the coupled structure parameter and growing networks from simple to complex ones progressively. Experimental evidence shows that our method achieves comparable and even better performance than the competitive optimizers in exploring the sparse structure of several widely used backbones on the benchmark datasets. Remarkably, with early stopping, our method unveils "winning tickets" in early epochs: the effective sparse network structures with comparable test accuracy to fully trained over-parameterized models, that are further transferable to similar alternative tasks. Furthermore, our method is able to grow networks efficiently with adaptive filter configurations, demonstrating the good performance with much less computational cost. Codes and models can be downloaded at https://github.com/DessiLBI2020/DessiLBI.

A Multiobjective Sparse Feature Learning Model for Deep Neural Networks

Fuzzy Multi-objective Sparse Feature Learning

Structure Learning for Deep Neural Networks Based on Multiobjective Optimization

Sparse Restricted Boltzmann Machine Based on Multiobjective Optimization.

Solving Large-Scale Multiobjective Optimization Problems With Sparse Optimal Solutions via Unsupervised Neural Networks

A Deep Learning Method Combined Sparse Autoencoder with SVM

On Adaptive Learning Framework for Deep Weighted Sparse Autoencoder: A Multiobjective Evolutionary Algorithm

Heterogeneous Feature Selection with Multi-Modal Deep Neural Networks and Sparse Group LASSO.

Feature Ensemble Learning Based on Sparse Autoencoders for Image Classification

Representation Learning with Deep Sparse Auto-Encoder for Multi-Task Learning

Sparse Feature Learning of Hyperspectral Imagery Via Multiobjective-Based Extreme Learning Machine

Parallel Design of Sparse Deep Belief Network with Multi-Objective Optimization

Deep subspace clustering to achieve jointly latent feature extraction and discriminative learning

An Empirical Analysis of Different Sparse Penalties for Autoencoder in Unsupervised Feature Learning

Objective-hierarchy Based Large-Scale Evolutionary Algorithm for Improving Joint Sparsity-Compression of Neural Network.

An efficient evolutionary algorithm based on deep reinforcement learning for large-scale sparse multiobjective optimization

Hybrid Embedded Deep Stacked Sparse Autoencoder with w_LPPD SVM Ensemble

Multi-regularization sparse reconstruction based on multifactorial multiobjective optimization

A Novel Deep Learning Approach: Stacked Evolutionary Auto-Encoder

Exploring Structural Sparsity of Deep Networks Via Inverse Scale Spaces

Hierarchical Sparse Method with Applications in Vision and Speech Recognition