Abstract:Despite the theoretical superiority in convergence issues, second-order optimizers are generally not among the top choices for training large-scale neural networks due to their high computational and memory cost. Nevertheless, introduced in recent progress of parameter-efficient tuning is a new paradigm that large-scale pre-trained models (PTMs) can be adapted to specific tasks by optimizing a tiny proportion of parameters, which might hopefully change the game. We associate this new paradigm with the computational tractability of second-order optimizers and succeed in applying them to large PTMs that are from hundreds of millions to billions in scale. Beyond verifying their tractability, we further investigate the stability-influencing factors in the optimization process and propose accordingly a Newton-step-clipping approach in which we clip the update tensors rather than the gradients. This approach stabilizes the convergence by gating the magnitude of Newton steps along the optimization trajectories through the rugged landscapes of deep transformers. We conduct extensive experiments across different downstream tasks, demonstrating that, when equipped with our Newton-step-clipping strategy, second-order optimizers, especially Kronecker-factored curvature approximation (K-FAC), can attain comparable and even superior results and faster convergence to those state-of-the-art bars implemented with AdamW. Furthermore, we scale the model up to 3 billion parameters and validate the tractability and effectiveness of our method. This work is not only the first successful application of second-order optimization on such large-scale models but also sheds light on the possibility of further optimization-wise analysis on large-scale models in the future.

Deep Parameter Optimisation

Generalized Parameter Estimation Method for Model-Based Real Time Optimization

Parameter Efficient Fine Tuning: A Comprehensive Analysis Across Applications

Parallel computing based parameter auto-tuning algorithm for optimization solvers

A Fast Parameter Tuning Framework Via Transfer Learning and Multi-Objective Bayesian Optimization

PATSMA: Parameter Auto-tuning for Shared Memory Algorithms

See Further for Parameter Efficient Fine-tuning by Standing on the Shoulders of Decomposition

Stage-based Hyper-parameter Optimization for Deep Learning

A Spark Optimizer for Adaptive, Fine-Grained Parameter Tuning

Hyper-Tune: Towards Efficient Hyper-parameter Tuning at Scale

Applying Second Order Optimization to Deep Transformers with Parameter-Efficient Tuning

Parameter Tuning for Self-optimizing Software at Scale

Pass-Tuning: Towards Structure-Aware Parameter-Efficient Tuning for Code Representation Learning

Delving into Parameter-Efficient Fine-Tuning in Code Change Learning: an Empirical Study

MindOpt Tuner: Boost the Performance of Numerical Software by Automatic Parameter Tuning

A novel method for real parameter optimization based on Gene Expression Programming

FastTuning: Enabling Fast and Efficient Hyper-Parameter Tuning with Partitioning and Parallelism of Search Space

Three Diverse Applications of General-Purpose Parameter Optimization Algorithm

Impact of Parameter Tuning for Optimizing Deep Neural Network Models for Predicting Software Faults