Abstract:Modern deep convolutional neural networks(CNNs) are often designed to be scalable, leading to the model family concept. A model family is a large (possibly infinite) collection of related neural network architectures. The isomorphism of a model family refers to the fact that the models within it share the same high-level structure. Meanwhile, the models within the model family are called isomorphic models for each other. Existing weight initialization methods for CNNs use random initialization or data-driven initialization. Even though these methods can perform satisfactory initialization, the isomorphism of model families is rarely explored. This work proposes an isomorphic model-based initialization method (IM Init) for CNNs. It can initialize any network with another well-trained isomorphic model in the same model family. We first formulate the widely used general network structure of CNNs. Then a structural weight transformation is presented to transform the weight between two isomorphic models. Finally, we apply our IM Init to the model down-sampling and up-sampling scenarios and confirm its effectiveness in improving accuracy and convergence speed through experiments on various image classification datasets. In the model down-sampling scenario, IM Init initializes the smaller target model with a larger well-trained source model. It improves the accuracy of RegNet200MF by 1.59% on the CIFAR-100 dataset and 1.9% on the CUB200 dataset. Inversely, IM Init initializes the larger target model with a smaller well-trained source model in the model up-sampling scenario. It significantly speeds up the convergence of RegNet600MF and improves the accuracy by 30.10% under short training schedules. Code will be available.

Regularization and Iterative Initialization of Softmax for Fast Training of Convolutional Neural Networks.

Batch-Normalization-based Soft Filter Pruning for Deep Convolutional Neural Networks

An Effective Training Method for Deep Convolutional Neural Network

Selective Output Smoothing Regularization: Regularize Neural Networks by Softening Output Distributions

Noisy Softmax: Improving the Generalization Ability of DCNN Via Postponing the Early Softmax Saturation

Stop-Gradient Softmax Loss for Deep Metric Learning.

Improving Convolutional Neural Networks Via Compacting Features

$\mathcal{G}$-softmax: Improving Intra-class Compactness and Inter-class Separability of Features

Learning deep discriminative embeddings via joint rescaled features and log-probability centers

Softmax Dissection: Towards Understanding Intra- and Inter-class Objective for Embedding Learning

Class-Variant Margin Normalized Softmax Loss for Deep Face Recognition

Contraction Mapping of Feature Norms for Classifier Learning on the Data with Different Quality

Convolutional Normalization: Improving Deep Convolutional Network Robustness and Training

Imbalance Robust Softmax for Deep Embeeding Learning

Improving Classification Performance of Softmax Loss Function Based on Scalable Batch-Normalization

Improvement of Pose Recognition by Sparse Regularized Convolutional Neural Network.

On the Learning Property of Logistic and Softmax Losses for Deep Neural Networks

Isomorphic Model-Based Initialization for Convolutional Neural Networks

Contraction Mapping of Feature Norms for Data Quality Imbalance Learning

Regularizing Deep Convolutional Neural Networks with a Structured Decorrelation Constraint.

Regularized Deep Transfer Learning: when CNN Meets Knn.