Abstract:Incremental learning aims to adapt to new sets of categories over time with minimal computational overhead. Prior work often addresses this task by training efficient task-specific adaptors that modify frozen layer weights or features to capture relevant information without affecting predictions on previously learned categories. While these adaptors are generally more efficient than finetuning the entire network, they still require tens to hundreds of thousands of task-specific trainable parameters even for relatively small networks, making it challenging to operate on resource-constrained environments with high communication costs like edge devices or mobile phones. Thus, we propose Reparameterized, Compact weight Adaptation for Sequential Tasks (RECAST), a novel method that dramatically reduces task-specific trainable parameters to fewer than 50 - several orders of magnitude less than competing methods like LoRA. RECAST accomplishes this efficiency by learning to decompose layer weights into a soft parameter-sharing framework consisting of shared weight templates and very few module-specific scaling factors or coefficients. This soft parameter-sharing framework allows for effective task-wise reparameterization by tuning only these coefficients while keeping templates frozen.A key innovation of RECAST is the novel weight reconstruction pipeline called Neural Mimicry, which eliminates the need for pretraining from scratch. This allows for high-fidelity emulation of existing pretrained weights within our framework and provides quick adaptability to any model scale and architecture. Extensive experiments across six datasets demonstrate RECAST outperforms the state-of-the-art by up to 3% across various scales, architectures, and parameter spaces Moreover, we show that RECAST's architecture-agnostic nature allows for seamless integration with existing methods, further boosting performance.

DyRep: Bootstrapping Training with Dynamic Re-parameterization

Efficient Re-parameterization Operations Search for Easy-to-Deploy Network Based on Directional Evolutionary Strategy

RepSR: Training Efficient VGG-style Super-Resolution Networks with Structural Re-Parameterization and Batch Normalization

RepAn: Enhanced Annealing Through Re-parameterization

RepGhost: A Hardware-Efficient Ghost Module via Re-parameterization

PaReprop: Fast Parallelized Reversible Backpropagation

Rethinking the Usage of Batch Normalization and Dropout in the Training of Deep Neural Networks

Towards Efficient Visual Adaption via Structural Re-parameterization

SepRep-Net: Multi-source Free Domain Adaptation via Model Separation And Reparameterization

RepAct: The Re-parameterizable Adaptive Activation Function

Post-training quantization for re-parameterization via coarse & fine weight splitting

Expanding and Refining Hybrid Compressors for Efficient Object Re-Identification

Reparameterization Head for Efficient Multi-Input Networks.

Η -Repyolo: Real-Time Object Detection Method Based on Η -Repconv and YOLOv8

RECAST: Reparameterized, Compact weight Adaptation for Sequential Tasks

PAD-Net: An Efficient Framework for Dynamic Networks

Dyn-Adapter: Towards Disentangled Representation for Efficient Visual Recognition

Dynamic Weighting Network for Person Re-Identification

Enhancing Accuracy and Parameter-Efficiency of Neural Representations for Network Parameterization

PYRA: Parallel Yielding Re-Activation for Training-Inference Efficient Task Adaptation