Abstract:Weight play an essential role in deep learning network models. Unlike network structure design, this article proposes the concept of weight augmentation, focusing on weight exploration. The core of Weight Augmentation Strategy (WAS) is to adopt random transformed weight coefficients training and transformed coefficients, named Shadow Weight(SW), for networks that can be used to calculate loss function to affect parameter updates. However, stochastic gradient descent is applied to Plain Weight(PW), which is referred to as the original weight of the network before the random transformation. During training, numerous SW collectively form high-dimensional space, while PW is directly learned from the distribution of SW instead of the data. The weight of the accuracy-oriented mode(AOM) relies on PW, which guarantees the network is highly robust and accurate. The desire-oriented mode(DOM) weight uses SW, which is determined by the network model's unique functions based on WAT's performance desires, such as lower computational complexity, lower sensitivity to particular data, etc. The dual mode be switched at anytime if needed. WAT extends the augmentation technique from data augmentation to weight, and it is easy to understand and implement, but it can improve almost all networks amazingly. Our experimental results show that convolutional neural networks, such as VGG-16, ResNet-18, ResNet-34, GoogleNet, MobilementV2, and Efficientment-Lite, can benefit much at little or no cost. The accuracy of models is on the CIFAR100 and CIFAR10 datasets, which can be evaluated to increase by 7.32\% and 9.28\%, respectively, with the highest values being 13.42\% and 18.93\%, respectively. In addition, DOM can reduce floating point operations (FLOPs) by up to 36.33\%. The code is available at <a class="link-external link-https" href="https://github.com/zlearh/Weight-Augmentation-Technology" rel="external noopener nofollow">this https URL</a>.

Don't Wait, Just Weight: Improving Unsupervised Representations by Learning Goal-Driven Instance Weights

Learning to Auto Weight: Entirely Data-Driven and Highly Efficient Weighting Framework

DAW: Exploring the Better Weighting Function for Semi-supervised Semantic Segmentation.

Auxiliary Task Reweighting for Minimum-data Learning

Sample Weighting: an Inherent Approach for Outlier Suppressing Discriminant Analysis

Importance Weighting Can Help Large Language Models Self-Improve

Gradient-based Class Weighting for Unsupervised Domain Adaptation in Dense Prediction Visual Tasks

Self-Tuning for Data-Efficient Deep Learning

Improving the Accuracy of Learning Example Weights for Imbalance Classification

On Improving the Algorithm-, Model-, and Data- Efficiency of Self-Supervised Learning

Weighted-learning-instance-based Retrieval Model Using Instance Distance.

Unsupervised Representation Learning by Balanced Self Attention Matching

Weights Augmentation: it has never ever ever ever let her model down

Debiased Self-Training for Semi-Supervised Learning

Reliable Weighted Optimal Transport For Unsupervised Domain Adaptation

Learning to Reweight with Deep Interactions

WOTBoost: Weighted Oversampling Technique in Boosting for imbalanced learning

DaWin: Training-free Dynamic Weight Interpolation for Robust Adaptation

IW-GAE: Importance Weighted Group Accuracy Estimation for Improved Calibration and Model Selection in Unsupervised Domain Adaptation

Trainable Weight Averaging for Fast Convergence and Better Generalization