Abstract:In low-latency or mobile applications, lower computation complexity, lower memory footprint and better energy efficiency are desired. Many prior works address this need by removing redundant parameters. Parameter quantization replaces floating-point arithmetic with lower precision fixed-point arithmetic, further reducing complexity. Typical training of quantized weight neural networks starts from fully quantized weights. Quantization creates random noise. As a way to compensate for this noise, during training, we propose to quantize some weights while keeping others in floating-point precision. A deep neural network has many layers. To arrive at a fully quantized weight network, we start from one quantized layer and then quantize more and more layers. We show that the order of layer quantization affects accuracies. Order count is large for deep neural networks. A sensitivity pre-training is proposed to guide the layer quantization order. Recent work in weight binarization replaces weight-input matrix multiplication with additions. We apply the proposed iterative training to weight binarization. Our experiments cover fully connected and convolutional networks on MNIST, CIFAR-10 and ImageNet datasets. We show empirically that, starting from partial binary weights instead of from fully binary ones, training reaches fully binary weight networks with better accuracies for larger and deeper networks. Layer binarization in the forward order results in better accuracies. Guided layer binarization can further improve that. The improvements come at a cost of longer training time.

Training Ternary Neural Networks by Rectified L2 Regularization

Optimizing Quantized Neural Networks in a Weak Curvature Manifold

Structured Deep Neural Network Pruning by Varying Regularization Parameters.

Simultaneously Optimizing Weight and Quantizer of Ternary Neural Network using Truncated Gaussian Approximation

Network as Regularization for Training Deep Neural Networks: Framework, Model and Performance

Learning Sparse & Ternary Neural Networks with Entropy-Constrained Trained Ternarization (EC2T)

Improving Low-Precision Network Quantization via Bin Regularization

BitNet: Bit-Regularized Deep Neural Networks

Temperature Balancing, Layer-wise Weight Analysis, and Neural Network Training

Regularizing Deep Networks Using Efficient Layerwise Adversarial Training

Improve Generalization and Robustness of Neural Networks via Weight Scale Shifting Invariant Regularizations

Iterative Training: Finding Binary Weight Deep Neural Networks with Layer Binarization

How (Implicit) Regularization of ReLU Neural Networks Characterizes the Learned Function -- Part II: the Multi-D Case of Two Layers with Random First Layer

Training Compact DNNs with l 1 / 2 Regularization

EraseReLU: A Simple Way to Ease the Training of Deep Convolution Neural Networks.

Effective Neural Network $L_0$ Regularization With BinMask

Ternary Weight Networks

Training a Two Layer ReLU Network Analytically

Neighborhood Region Smoothing Regularization for Finding Flat Minima in Deep Neural Networks

A Partial Regularization Method for Network Compression

Recurrent Neural Network Regularization