Abstract:Inference for Deep Neural Networks is increasingly being executed locally on mobile and embedded platforms due to its advantages in latency, privacy and connectivity. Since modern System on Chips typically execute a combination of different and dynamic workloads concurrently, it is challenging to consistently meet inference time/energy budget at runtime because of the local computing resources available to the DNNs vary considerably. To address this challenge, a variety of dynamic DNNs were proposed. However, these works have significant memory overhead, limited runtime recoverable compression rate and narrow dynamic ranges of performance scaling. In this paper, we present a dynamic DNN using incremental training and group convolution pruning. The channels of the DNN convolution layer are divided into groups, which are then trained incrementally. At runtime, following groups can be pruned for inference time/energy reduction or added back for accuracy recovery without model retraining. In addition, we combine task mapping and Dynamic Voltage Frequency Scaling (DVFS) with our dynamic DNN to deliver finer trade-off between accuracy and time/power/energy over a wider dynamic range. We illustrate the approach by modifying AlexNet for the CIFAR10 image dataset and evaluate our work on two heterogeneous hardware platforms: Odroid XU3 (ARM <a class="link-external link-http" href="http://big.LITTLE" rel="external noopener nofollow">this http URL</a> CPUs) and Nvidia Jetson Nano (CPU and GPU). Compared to the existing works, our approach can provide up to 2.36x (energy) and 2.73x (time) wider dynamic range with a 2.4x smaller memory footprint at the same compression rate. It achieved 10.6x (energy) and 41.6x (time) wider dynamic range by combining with task mapping and DVFS.

DACO: Pursuing Ultra-low Power Consumption Via DNN-Adaptive CPU-GPU CO-optimization on Mobile Devices

MOC: Multi-Objective Mobile CPU-GPU Co-Optimization for Power-Efficient DNN Inference

Explore Training of Deep Convolutional Neural Networks on Battery-powered Mobile Devices: Design and Application

Survey on Energy-Efficient Deep Neural Networks for Computer Vision

All-in-One: A Highly Representative DNN Pruning Framework for Edge Devices with Dynamic Power Management

On-Demand Deep Model Compression for Mobile Devices

CODA: Improving Resource Utilization by Slimming and Co-locating DNN and CPU Jobs

AdaOper: Energy-efficient and Responsive Concurrent DNN Inference on Mobile Devices

Mobile or FPGA? A Comprehensive Evaluation on Energy Efficiency and a Unified Optimization Framework

Towards Ultra-High Performance and Energy Efficiency of Deep Learning Systems: An Algorithm-Hardware Co-Optimization Framework

Exploiting Neural-Network Statistics for Low-Power DNN Inference

Optimizing DNN Inference on Multi-Accelerator SoCs at Training-time

DPP-Net: Device-Aware Progressive Search for Pareto-Optimal Neural Architectures

Double-Shift: A Low-Power DNN Weights Storage and Access Framework Based on Approximate Decomposition and Quantization

Data Overfitting for On-Device Super-Resolution with Dynamic Algorithm and Compiler Co-Design

Joint Optimization of Data Transfer and Co-Execution for DNN in Edge Computing

Incremental Training and Group Convolution Pruning for Runtime DNN Performance Scaling on Heterogeneous Embedded Platforms

AdaScale: Dynamic Context-aware DNN Scaling via Automated Adaptation Loop on Mobile Devices

Power and Latency Collaborative Optimizations of Object Detection CNN on Edge Devices under Limited Energy Budget

A Comprehensive Analysis of Low-Impact Computations in Deep Learning Workloads

An Overview of Energy-Efficient Hardware Accelerators for On-Device Deep-Neural-Network Training