Abstract:To facilitate the evolution of edge intelligence in ever-changing environments, we study on-device incremental learning constrained in limited computation resource in this paper. Current on-device training methods just focus on efficient training without considering the catastrophic forgetting, preventing the model getting stronger when continually exploring the world. To solve this problem, a direct solution is to involve the existing incremental learning mechanisms into the on-device training framework. Unfortunately, such a manner cannot work well as those mechanisms usually introduce large additional computational cost to the network optimization process, which would inevitably exceed the memory capacity of the edge devices. To address this issue, this paper makes an early effort to propose a simple but effective edge-friendly incremental learning framework. Based on an empirical study on the knowledge intensity of the kernel elements of the neural network, we find that the center kernel is the key for maximizing the knowledge intensity for learning new data, while freezing the other kernel elements would get a good balance on the model's capacity for overcoming catastrophic forgetting. Upon this finding, we further design a center-sensitive kernel optimization framework to largely alleviate the cost of the gradient computation and back-propagation. Besides, a dynamic channel element selection strategy is also proposed to facilitate a sparse orthogonal gradient projection for further reducing the optimization complexity, upon the knowledge explored from the new task data. Extensive experiments validate our method is efficient and effective, e.g., our method achieves average accuracy boost of 38.08% with even less memory and approximate computation compared to existing on-device training methods, indicating its significant potential for on-device incremental learning.

Efficient On-Device Incremental Learning by Weight Freezing.

Center-Sensitive Kernel Optimization for Efficient On-Device Incremental Learning

Incremental Learning Via Rate Reduction

A Class Incremental Learning Method with Weight Re-Initialization and Multiple Distillation for Image Classification

Enabling Deep Learning on Edge Devices through Filter Pruning and Knowledge Transfer

Learning with Recoverable Forgetting

Incremental Learning Using Conditional Adversarial Networks

Maintaining Discrimination and Fairness in Class Incremental Learning

Incremental Learning In Online Scenario

Memory-efficient Deep Learning Inference with Incremental Weight Loading and Data Layout Reorganization on Edge Systems.

Efficient Continual Learning with Low Memory Footprint For Edge Device

Cache and Reuse: Rethinking the Efficiency of On-device Transfer Learning

Accelerating Deep Learning Inference via Freezing

Fast Deep Learning Training Through Intelligently Freezing Layers

Rethinking Class-Incremental Learning from a Dynamic Imbalanced Learning Perspective

Neuromorphic Incremental on-chip Learning with Hebbian Weight Consolidation

Model Behavior Preserving for Class-Incremental Learning

Overcoming Memory Constraint for Improved Target Classification Performance on Embedded Deep Learning Systems

A Continual and Incremental Learning Approach for TinyML On-device Training Using Dataset Distillation and Model Size Adaption

An Efficient Replay for Class-Incremental Learning with Pre-trained Models

Adaptive knowledge transfer for class incremental learning