Judging Adam: Studying the Performance of Optimization Methods on ML4SE Tasks

Dmitry Pasechnyuk,Anton Prazdnichnykh,Mikhail Evtikhiev,Timofey Bryksin

2023-03-07

Abstract:Solving a problem with a deep learning model requires researchers to optimize the loss function with a certain optimization method. The research community has developed more than a hundred different optimizers, yet there is scarce data on optimizer performance in various tasks. In particular, none of the benchmarks test the performance of optimizers on source code-related problems. However, existing benchmark data indicates that certain optimizers may be more efficient for particular domains. In this work, we test the performance of various optimizers on deep learning models for source code and find that the choice of an optimizer can have a significant impact on the model quality, with up to two-fold score differences between some of the relatively well-performing optimizers. We also find that RAdam optimizer (and its modification with the Lookahead envelope) is the best optimizer that almost always performs well on the tasks we consider. Our findings show a need for a more extensive study of the optimizers in code-related tasks, and indicate that the ML4SE community should consider using RAdam instead of Adam as the default optimizer for code-related deep learning tasks.

Software Engineering,Machine Learning

What problem does this paper attempt to address?

The problem that this paper attempts to solve is the impact of optimizer selection on model performance in machine - learning - for - software - engineering tasks (ML4SE) in software engineering. Although most researchers and practitioners currently use the Adam optimizer by default in the ML4SE field, there is a lack of performance data on different optimizers for these specific tasks. Therefore, this paper aims to evaluate the impact of optimizer selection on model prediction quality by testing the performance of multiple optimizers on different deep - learning models and explore whether there are optimizers more suitable for code - related tasks.

Specifically, the researchers selected four models (CODE2SEQ, TREE - LSTM, CODEGNN, CODETRANSFORMER) and two code - to - text generation tasks (document generation and method name generation) and conducted experiments on multiple datasets. The experimental results show that the choice of optimizer does significantly affect the performance of the model, and some optimizers such as RAdam (and its Lookahead variant) almost always perform well in the considered tasks. This indicates that for code - related deep - learning tasks, the ML4SE community should consider using RAdam instead of Adam as the default optimizer.

In addition, the study also found that the performance of the optimizer depends not only on the dataset and the task, but also on the model architecture. Therefore, the paper suggests that researchers and practitioners try multiple optimizers when developing models to improve the quality of the models.