Abstract:Reducing the memory footprint of Machine Learning (ML) models, particularly Deep Neural Networks (DNNs), is essential to enable their deployment into resource-constrained tiny devices. However, a disadvantage of DNN models is their vulnerability to adversarial attacks, as they can be fooled by adding slight perturbations to the inputs. Therefore, the challenge is how to create accurate, robust, and tiny DNN models deployable on resource-constrained embedded devices. This paper reports the results of devising a tiny DNN model, robust to adversarial black and white box attacks, trained with an automatic quantizationaware training framework, i.e. QKeras, with deep quantization loss accounted in the learning loop, thereby making the designed DNNs more accurate for deployment on tiny devices. We investigated how QKeras and an adversarial robustness technique, Jacobian Regularization (JR), can provide a co-optimization strategy by exploiting the DNN topology and the per layer JR approach to produce robust yet tiny deeply quantized DNN models. As a result, a new DNN model implementing this cooptimization strategy was conceived, developed and tested on three datasets containing both images and audio inputs, as well as compared its performance with existing benchmarks against various white-box and black-box attacks. Experimental results demonstrated that on average our proposed DNN model resulted in 8.3% and 79.5% higher accuracy than MLCommons/Tiny benchmarks in the presence of white-box and black-box attacks on the CIFAR-10 image dataset and a subset of the Google Speech Commands audio dataset respectively. It was also 6.5% more accurate for black-box attacks on the SVHN image dataset.

Error-Silenced Quantization: Bridging Robustness and Compactness

Pse: Mixed Quantization Framework of Neural Networks for Efficient Deployment

Defensive Quantization: When Efficiency Meets Robustness

Hessian-based Mixed-Precision Quantization with Transition Aware Training for Neural Networks

Double-Win Quant: Aggressively Winning Robustness of Quantized Deep Neural Networks Via Random Precision Training and Inference

Error-aware Quantization through Noise Tempering

Benchmarking the Robustness of Quantized Models

RobustMQ: Benchmarking Robustness of Quantized Models

Improving the Robustness of Quantized Deep Neural Networks to White-Box Attacks using Stochastic Quantization and Information-Theoretic Ensemble Training

Improving Robustness Against Adversarial Attacks with Deeply Quantized Neural Networks

Investigating the Impact of Quantization on Adversarial Robustness

A Layer-wise Adversarial-aware Quantization Optimization for Improving Robustness

RAND: Robustness Aware Norm Decay For Quantized Seq2seq Models

Deep Network Quantization via Error Compensation

An Integrated Approach to Produce Robust Models with High Efficiency

LQ-Nets: Learned Quantization for Highly Accurate and Compact Deep Neural Networks

Nearest is Not Dearest: Towards Practical Defense against Quantization-conditioned Backdoor Attacks

Bit-shrinking: Limiting Instantaneous Sharpness for Improving Post-training Quantization

Robustness of Neural Networks to Parameter Quantization

Efficient Neural Compression with Inference-time Decoding

Robust Training of Neural Networks at Arbitrary Precision and Sparsity