Abstract:To facilitate the evolution of edge intelligence in ever-changing environments, we study on-device incremental learning constrained in limited computation resource in this paper. Current on-device training methods just focus on efficient training without considering the catastrophic forgetting, preventing the model getting stronger when continually exploring the world. To solve this problem, a direct solution is to involve the existing incremental learning mechanisms into the on-device training framework. Unfortunately, such a manner cannot work well as those mechanisms usually introduce large additional computational cost to the network optimization process, which would inevitably exceed the memory capacity of the edge devices. To address this issue, this paper makes an early effort to propose a simple but effective edge-friendly incremental learning framework. Based on an empirical study on the knowledge intensity of the kernel elements of the neural network, we find that the center kernel is the key for maximizing the knowledge intensity for learning new data, while freezing the other kernel elements would get a good balance on the model's capacity for overcoming catastrophic forgetting. Upon this finding, we further design a center-sensitive kernel optimization framework to largely alleviate the cost of the gradient computation and back-propagation. Besides, a dynamic channel element selection strategy is also proposed to facilitate a sparse orthogonal gradient projection for further reducing the optimization complexity, upon the knowledge explored from the new task data. Extensive experiments validate our method is efficient and effective, e.g., our method achieves average accuracy boost of 38.08% with even less memory and approximate computation compared to existing on-device training methods, indicating its significant potential for on-device incremental learning.

Efficient Continual Learning with Low Memory Footprint For Edge Device

TinyCL: An Efficient Hardware Architecture for Continual Learning on Autonomous Systems

Memory-efficient Deep Learning Inference with Incremental Weight Loading and Data Layout Reorganization on Edge Systems.

Overcoming Memory Constraint for Improved Target Classification Performance on Embedded Deep Learning Systems

Efficient Knowledge Management for Heterogeneous Federated Continual Learning on Resource-Constrained Edge Devices

Resource-Efficient Heterogenous Federated Continual Learning on Edge

Center-Sensitive Kernel Optimization for Efficient On-Device Incremental Learning

Efficient knowledge management for heterogenous federated continual learning on resource-constrained edge devices

LifeLearner: Hardware-Aware Meta Continual Learning System for Embedded Computing Platforms

Mixed‐Precision Continual Learning Based on Computational Resistance Random Access Memory

FlexNN: Efficient and Adaptive DNN Inference on Memory-Constrained Edge Devices.

Continual Deep Learning on the Edge via Stochastic Local Competition among Subnetworks

A TinyML Platform for On-Device Continual Learning With Quantized Latent Replays

Continual Learning in the Frequency Domain

Cross-FCL: Toward a Cross-edge Federated Continual Learning Framework in Mobile Edge Computing Systems

LR-CNN: Lightweight Row-centric Convolutional Neural Network Training for Memory Reduction

Schematic Memory Persistence and Transience for Efficient and Robust Continual Learning

etuner: A Redundancy-Aware Framework for Efficient Continual Learning Application on Edge Devices

Adaptive Progressive Continual Learning.

ECLM: Efficient Edge-Cloud Collaborative Learning with Continuous Environment Adaptation

NeuLite: Memory-Efficient Federated Learning via Elastic Progressive Training