Abstract:Modern deep convolutional neural networks(CNNs) are often designed to be scalable, leading to the model family concept. A model family is a large (possibly infinite) collection of related neural network architectures. The isomorphism of a model family refers to the fact that the models within it share the same high-level structure. Meanwhile, the models within the model family are called isomorphic models for each other. Existing weight initialization methods for CNNs use random initialization or data-driven initialization. Even though these methods can perform satisfactory initialization, the isomorphism of model families is rarely explored. This work proposes an isomorphic model-based initialization method (IM Init) for CNNs. It can initialize any network with another well-trained isomorphic model in the same model family. We first formulate the widely used general network structure of CNNs. Then a structural weight transformation is presented to transform the weight between two isomorphic models. Finally, we apply our IM Init to the model down-sampling and up-sampling scenarios and confirm its effectiveness in improving accuracy and convergence speed through experiments on various image classification datasets. In the model down-sampling scenario, IM Init initializes the smaller target model with a larger well-trained source model. It improves the accuracy of RegNet200MF by 1.59% on the CIFAR-100 dataset and 1.9% on the CUB200 dataset. Inversely, IM Init initializes the larger target model with a smaller well-trained source model in the model up-sampling scenario. It significantly speeds up the convergence of RegNet600MF and improves the accuracy by 30.10% under short training schedules. Code will be available.

How to Initialize your Network? Robust Initialization for WeightNorm & ResNets

Deep Residual Networks and Weight Initialization

A Sober Look at Neural Network Initializations

Identical Initialization: A Universal Approach to Fast and Stable Training of Neural Networks

A mathematical framework for improved weight initialization of neural networks using Lagrange multipliers

A weight initialization based on the linear product structure for neural networks

Improved weight initialization for deep and narrow feedforward neural network

On weight initialization in deep neural networks

Advancing Neural Network Performance through Emergence-Promoting Initialization Scheme

Rethinking the Usage of Batch Normalization and Dropout in the Training of Deep Neural Networks

Weight initialization based‐rectified linear unit activation function to improve the performance of a convolutional neural network model

Robust Implicit Regularization via Weight Normalization

Critical Initialization of Wide and Deep Neural Networks through Partial Jacobians: General Theory and Applications

Neuron Campaign for Initialization Guided by Information Bottleneck Theory

Isomorphic Model-Based Initialization for Convolutional Neural Networks

Initialization Seeds Facilitating Neural Network Quantization

Provable Benefit of Orthogonal Initialization in Optimizing Deep Linear Networks

Projection Based Weight Normalization for Deep Neural Networks.

On Using Quasirandom Sequences in Machine Learning for Model Weight Initialization

Where Should We Begin? A Low-Level Exploration of Weight Initialization Impact on Quantized Behaviour of Deep Neural Networks

Batch Normalization Biases Residual Blocks Towards the Identity Function in Deep Networks