Abstract:Quantized deep neural networks (QDNNs) are attractive due to their much lower memory storage and faster inference speed than their regular full-precision counterparts. To maintain the same performance level especially at low bit-widths, QDNNs must be retrained. Their training involves piecewise constant activation functions and discrete weights; hence, mathematical challenges arise. We introduce the notion of coarse gradient and propose the blended coarse gradient descent (BCGD) algorithm, for training fully quantized neural networks. Coarse gradient is generally not a gradient of any function but an artificial ascent direction. The weight update of BCGD goes by coarse gradient correction of a weighted average of the full-precision weights and their quantization (the so-called blending), which yields sufficient descent in the objective value and thus accelerates the training. Our experiments demonstrate that this simple blending technique is very effective for quantization at extremely low bit-width such as binarization. In full quantization of ResNet-18 for ImageNet classification task, BCGD gives 64.36% top-1 accuracy with binary weights across all layers and 4-bit adaptive activation. If the weights in the first and last layers are kept in full precision, this number increases to 65.46%. As theoretical justification, we show convergence analysis of coarse gradient descent for a two-linear-layer neural network model with Gaussian input data and prove that the expected coarse gradient correlates positively with the underlying true gradient.

Modified quasi-Newton algorithm for training large-scale feedforward neural networks and its application

Cuckoo Search Algorithm with Membrane Communication Mechanism for Modeling Overhead Crane Systems Using RBF Neural Networks.

Mini-batch Quasi-Newton Optimization for Large Scale Linear Support Vector Regression

mL-BFGS: A Momentum-based L-BFGS for Distributed Large-Scale Neural Network Optimization

Quantum Algorithms for Feedforward Neural Networks

Practical Quasi-Newton Methods for Training Deep Neural Networks

New global learning algorithm for fuzzy neural networks based on modified quasi-newton method and genetic searching techniques

Fast Computational Approach to the Levenberg-Marquardt Algorithm for Training Feedforward Neural Networks

A L-BFGS Based Learning Algorithm for Complex-Valued Feedforward Neural Networks

Modelling of Electrohydraulic System Using RBF Neural Networks and Genetic Algorithm

A New Global Optimizing Algorithm for Fuzzy Neural Networks

QuadraNet V2: Efficient and Sustainable Training of High-Order Neural Networks with Quadratic Adaptation

A new hybrid conjugate gradient method close to the memoryless BFGS quasi-Newton method and its application in image restoration and machine learning

A Novel Global Training Algorithm and Its Convergence Theorem for Fuzzy Neural Networks

Efficient Vectorized Backpropagation Algorithms for Training Feedforward Networks Composed of Quadratic Neurons

Stochastic Quasi-Newton Optimization in Large Dimensions Including Deep Network Training

EA-CG: An Approximate Second-Order Method for Training Fully-Connected Neural Networks

Post-Training Quantization for Re-parameterization via Coarse & Fine Weight Splitting

Discrete Gradient-Zeroing Neural Network Algorithms for Handling Future Quadratic Program as well as Robot Arm via Ten-Instant Formula

Modelling of the Relaxation Least Squares-Based Neural Networks and Its Application

Blended coarse gradient descent for full quantization of deep neural networks