Abstract:In the past years, Deep convolution neural network has achieved great success in many artificial intelligence applications. However, its enormous model size and massive computation cost have become the main obstacle for deployment of such powerful algorithm in the low power and resource-limited mobile systems. As the countermeasure to this problem, deep neural networks with ternarized weights (i.e. -1, 0, +1) have been widely explored to greatly reduce the model size and computational cost, with limited accuracy degradation. In this work, we propose a novel ternarized neural network training method which simultaneously optimizes both weights and quantizer during training, differentiating from prior works. Instead of fixed and uniform weight ternarization, we are the first to incorporate the thresholds of weight ternarization into a closed-form representation using the truncated Gaussian approximation, enabling simultaneous optimization of weights and quantizer through back-propagation training. With both of the first and last layer ternarized, the experiments on the ImageNet classification task show that our ternarized ResNet-18/34/50 only has 3.9/2.52/2.16% accuracy degradation in comparison to the full-precision counterparts.
What problem does this paper attempt to address?
The problem that this paper attempts to solve is: in deep convolutional neural networks (DCNN), due to the huge model size and high computational cost, it is difficult to deploy these powerful algorithms in low - power - consumption and resource - limited mobile systems. To solve this problem, researchers have explored deep neural networks using ternary - valued weights (-1, 0, +1) to significantly reduce the model size and computational cost while minimizing the loss of accuracy.
Specifically, the paper proposes a novel training method for ternary - valued neural networks. During the training process, the weights and quantizers (i.e., thresholds) are optimized simultaneously, instead of using a fixed unified weight ternary - valued method. For the first time, the authors integrate the threshold of weight ternary - valued into a closed - form expression, using the truncated Gaussian approximation method, so that the weights and quantizers can be optimized simultaneously through back - propagation training.
The following are the key contributions of this paper:
1. **A fully trainable ternary - valued method for deep neural networks**: Jointly train the quantizer thresholds, inter - layer scaling factors and model weights to minimize the accuracy loss caused by model compression.
2. **An innovative threshold optimization method**: For the first time, integrate the threshold of weight ternary - valued into a closed - form expression, using the truncated Gaussian approximation method, so that the threshold can be optimized together with other network parameters through back - propagation.
3. **An improved straight - through estimator design**: Propose a new gradient correctness optimization method to provide a better gradient approximation for the non - differentiable step - like ternary - valued function, thereby speeding up the convergence rate and improving the inference accuracy.
Through these methods, the paper shows that on the ImageNet classification task, the ternary - valued ResNet - 18/34/50 models have only about 3.9%/2.52%/2.16% accuracy loss respectively compared to the full - precision models.
The key formulas represented in Markdown format are as follows:
\[
w^*,\delta^*=\arg\min_{w,\delta}L(y,f(x,w,\delta))
\]
where \(L\) is the defined network loss function, \(y\) is the target corresponding to the input tensor \(x\), and \(f(\cdot)\) calculates the network output with respect to the network parameters.
The weight ternary - valued function:
\[
w'_{l,i}=S_l(\mu_l,\sigma_l,\Delta_l)\cdot\text{Tern}(w_{l,i},\Delta_l)
=S_l(\mu_l,\sigma_l,\Delta_l)\cdot
\begin{cases}
+1&\text{if }w_{l,i}>\Delta^+_l\\
0&\text{if }\Delta^-_l\leq w_{l,i}\leq\Delta^+_l\\
-1&\text{if }w_{l,i}<\Delta^-_l
\end{cases}
\]
The closed - form expression of the inter - layer scaling factor \(S_l\):
\[
S_l(\mu_l,\sigma_l,\delta_l)=\mu_l+\sigma_l\cdot\frac{\phi(\alpha|0,1)}{1 - \Phi(\alpha|0,1)}
\]
where \(\alpha=\frac{\delta_l}{\sigma_l}\), and \(\phi(\cdot|0,1)\) and \(\Phi(\cdot|0,1)\) are the PDF and CDF of the standard normal distribution respectively.
Through these methods, the paper successfully solves the problem of significantly reducing the model size and computational complexity while maintaining high accuracy.