Abstract:The complexity of deep neural network algorithms for hardware implementation can be much lowered by optimizing the word-length of weights and signals. Direct quantization of floating-point weights, however, does not show good performance when the number of bits assigned is small. Retraining of quantized networks has been developed to relieve this problem. In this work, the effects of retraining are analyzed for a feedforward deep neural network (FFDNN) and a convolutional neural network (CNN). The network complexity is controlled to know their effects on the resiliency of quantized networks by retraining. The complexity of the FFDNN is controlled by varying the unit size in each hidden layer and the number of layers, while that of the CNN is done by modifying the feature map configuration. We find that the performance gap between the floating-point and the retrain-based ternary (+1, 0, -1) weight neural networks exists with a fair amount in 'complexity limited' networks, but the discrepancy almost vanishes in fully complex networks whose capability is limited by the training data, rather than by the number of connections. This research shows that highly complex DNNs have the capability of absorbing the effects of severe weight quantization through retraining, but connection limited networks are less resilient. This paper also presents the effective compression ratio to guide the trade-off between the network size and the precision when the hardware resource is limited.

Effects of Rounding-off Approximation on Quantization of Complex Networks

Quantization Effects on Complex Networks

Distributed fault estimation of networked systems using quantized measurements

On The Local Stochastic Stability Of Nonlinear Complex Networks

Nonfragile State Estimation of Quantized Complex Networks with Switching Topologies

Scaling Invariance in Wave Functions of Quantum Systems on Complex Networks

Structural Balance and Random Walks on Complex Networks with Complex Weights

Distributed Online Quantile Regression over Networks with Quantized Communication.

Complex networks with complex weights

The Effects Of Quantization On Multi-Layer Feedforward Neural Networks

The Quantization Effects of Different Probability Distribution on Multilayer Feedforward Neural Networks

Analysis of Quantization Effects on High-Order Function Neural Networks

On Expressive Power of Quantized Neural Networks under Fixed-Point Arithmetic

Towards the Limit of Network Quantization

Neural Network Quantization: Separate Scaling of Rows and Columns in Weight Matrix

Bussgang revisited: effect of quantization on signal to distortion plus noise ratio with non-Gaussian signals

Resiliency of Deep Neural Networks under Quantization

q Index Degree Distribution in Random Networks via Superstatistics

On the Effects of Quantisation on Model Uncertainty in Bayesian Neural Networks

Quantum Complexity of Weighted Diameter and Radius in CONGEST Networks

Efficient Adaptive Activation Rounding for Post-Training Quantization