Abstract:The implementation of machine learning in Internet of Things devices poses significant operational challenges due to limited energy and computation resources. In recent years, significant efforts have been made to implement simplified ML models that can achieve reasonable performance while reducing computation and energy, for example by pruning weights in neural networks, or using reduced precision for the parameters and arithmetic operations. However, this type of approach is limited by the performance of the ML implementation, i.e., by the loss for example in accuracy due to the model simplification. In this article, we present adaptive resolution inference (ARI), a novel approach that enables to evaluate new tradeoffs between energy dissipation and model performance in ML implementations. The main principle of the proposed approach is to run inferences with reduced precision (quantization) and use the margin over the decision threshold to determine if either the result is reliable, or the inference must run with the full model. The rationale is that quantization only introduces small deviations in the inference scores, such that if the scores have a sufficient margin over the decision threshold, it is unlikely that the full model would have a different result. Therefore, we can run the quantized model first, and only when the scores do not have a sufficient margin, the full model is run. This enables most inferences to run with the reduced precision model and only a small fraction requires the full model, so significantly reducing computation and energy while not affecting model performance. The proposed ARI approach is presented, analyzed in detail, and evaluated using different data sets for floating-point and stochastic computing implementations. The results show that ARI can significantly reduce the energy for inference in different configurations with savings between 40% and 85%.

EfficientBioAI: making bioimaging AI models efficient in energy and latency

EfficientBioAI: Making Bioimaging AI Models Efficient in Energy, Latency and Representation

Explore Training of Deep Convolutional Neural Networks on Battery-powered Mobile Devices: Design and Application

Survey on Energy-Efficient Deep Neural Networks for Computer Vision

Investigation of Energy-efficient AI Model Architectures and Compression Techniques for "Green" Fetal Brain Segmentation

EPAM: A Predictive Energy Model for Mobile AI

The importance of resource awareness in artificial intelligence for healthcare

A Survey on the Optimization of Neural Network Accelerators for Micro-AI On-Device Inference

Leveraging Computational Pathology AI for Noninvasive Optical Imaging Analysis Without Retraining

Breathing Life Into Artificial Intelligence*

Full-Cycle Energy Consumption Benchmark for Low-Carbon Computer Vision

Toward Cross-Layer Energy Optimizations in AI Systems

Toward Efficient Execution of Mainstream Deep Learning Frameworks on Mobile Devices: Architectural Implications

Deep learning based Image Compression for Microscopy Images: An Empirical Study

How to Collect and Interpret Medical Pictures Captured in Highly Challenging Environments that Range from Nanoscale to Hyperspectral Imaging

AIMIC: Deep Learning for Microscopic Image Classification

Adaptive Resolution Inference (ARI): Energy-Efficient Machine Learning for Internet of Things

Implicit Neural Image Field for Biological Microscopy Image Compression

Estimating the Power Consumption of Heterogeneous Devices when performing AI Inference

Exploring AI-driven Innovations in Image Communication Systems for Enhanced Medical Imaging Applications

TinyM$^2$Net-V3: Memory-Aware Compressed Multimodal Deep Neural Networks for Sustainable Edge Deployment