Abstract:Pre-trained Artificial Neural Networks (ANNs) exhibit robust pattern recognition capabilities and share extensive similarities with the human brain, specifically Biological Neural Networks (BNNs). We are particularly intrigued by these models' ability to acquire new knowledge through fine-tuning. In this regard, Parameter-efficient Fine-tuning (PEFT) has gained widespread adoption as a substitute for full fine-tuning due to its cost reduction in training and mitigation of over-fitting risks by limiting the number of trainable parameters during adaptation. Since both ANNs and BNNs propagate information layer-by-layer, a common analogy can be drawn: weights in ANNs represent synapses in BNNs, while features (also known as latent variables or logits) in ANNs represent neurotransmitters released by neurons in BNNs. Mainstream PEFT methods aim to adjust feature or parameter values using only a limited number of trainable parameters (usually less than 1% of the total parameters), yet achieve surprisingly good results. Building upon this clue, we delve deeper into exploring the connections between feature adjustment and parameter adjustment, resulting in our proposed method Synapses & Neurons (SAN) that learns scaling matrices for features and propagates their effects towards posterior weight matrices. Our approach draws strong inspiration from well-known neuroscience phenomena - Long-term Potentiation (LTP) and Long-term Depression (LTD), which also reveal the relationship between synapse development and neurotransmitter release levels. We conducted extensive comparisons of PEFT on 26 datasets using attention-based networks as well as convolution-based networks, leading to significant improvements compared to other tuning methods (+8.5% over fully-finetune, +7% over Visual Prompt Tuning, and +3.2% over LoRA). The codes would be released.

Enhancing Accuracy and Parameter-Efficiency of Neural Representations for Network Parameterization

DyRep: Bootstrapping Training with Dynamic Re-parameterization

Expand-and-Cluster: Parameter Recovery of Neural Networks

Enhancing Deep Neural Network Training Efficiency and Performance through Linear Prediction

Parameter Prediction for Unseen Deep Architectures

Discovering Long-Term Effects on Parameter Efficient Fine-tuning

Weight Reparametrization for Budget-Aware Network Pruning

Adaptive inertia weights: an effective way to improve parameter estimation of hidden layer in stochastic configuration networks

Efficient and sparse neural networks by pruning weights in a multiobjective learning approach

Toward Compact Parameter Representations for Architecture-Agnostic Neural Network Compression

Post-Training Quantization for Re-parameterization via Coarse & Fine Weight Splitting

Training compact neural networks via

PELA: Learning Parameter-Efficient Models with Low-Rank Approximation

Towards a Unified View of Parameter-Efficient Transfer Learning

Training Compact Neural Networks via Auxiliary Overparameterization

Sequencing the Neurome: Towards Scalable Exact Parameter Reconstruction of Black-Box Neural Networks

NEAT: Nonlinear Parameter-efficient Adaptation of Pre-trained Models

Over Parameterized Two-level Neural Networks Can Learn Near Optimal Feature Representations

Sine Activated Low-Rank Matrices for Parameter Efficient Learning

Neural reparameterization improves structural optimization