Abstract:Neural networks are trained by minimizing a loss function that defines the discrepancy between the predicted model output and the target value. The selection of the loss function is crucial to achieve task-specific behaviour and highly influences the capability of the model. A variety of loss functions have been proposed for a wide range of tasks affecting training and model performance. For classification tasks, the cross entropy is the de-facto standard and usually the first choice. Here, we try to experimentally challenge the well-known loss functions, including cross entropy (CE) loss, by utilizing the genetic programming (GP) approach, a population-based evolutionary algorithm. GP constructs loss functions from a set of operators and leaf nodes and these functions are repeatedly recombined and mutated to find an optimal structure. Experiments were carried out on different small-sized datasets CIFAR-10, CIFAR-100 and Fashion-MNIST using an Inception model. The 5 best functions found were evaluated for different model architectures on a set of standard datasets ranging from 2 to 102 classes and very different sizes. One function, denoted as Next Generation Loss (NGL), clearly stood out showing same or better performance for all tested datasets compared to CE. To evaluate the NGL function on a large-scale dataset, we tested its performance on the Imagenet-1k dataset where it showed improved top-1 accuracy compared to models trained with identical settings and other losses. Finally, the NGL was trained on a segmentation downstream task for Pascal VOC 2012 and COCO-Stuff164k datasets improving the underlying model performance.
Computer Vision and Pattern Recognition,Machine Learning,Neural and Evolutionary Computing
What problem does this paper attempt to address?
The main objective of this paper is to explore a new method for designing loss functions and to experimentally verify its effectiveness in image classification and segmentation tasks. Specifically, the paper attempts to address the following key issues:
1. **Finding a universal and efficient loss function**: Traditionally, the choice of loss function (such as cross-entropy) is crucial for training deep learning models, but it often requires expert knowledge and extensive tuning. Therefore, the authors aim to automatically design a new loss function suitable for various datasets and model architectures through Genetic Programming (GP), an evolutionary algorithm.
2. **Improving model performance**: Although cross-entropy loss is widely used for classification tasks, the authors seek to find a new loss function that can surpass or at least match it, thereby enhancing the model's performance on different datasets.
3. **Adaptability and generalization ability**: The newly proposed loss function in the study, called Next Generation Loss (NGL), possesses self-regulating characteristics, meaning it can automatically adjust its behavior in different tasks, thereby improving the model's generalization ability.
To achieve these goals, the paper employs a Genetic Programming-based approach to search for loss functions. By conducting experiments on several small-scale datasets (such as CIFAR-10, CIFAR-100, and Fashion-MNIST), the researchers identified five optimal candidate loss functions. Among them, a loss function named NGL performed exceptionally well, showing superior or at least comparable performance to cross-entropy in classification tasks across multiple models (such as ResNet50 and InceptionV3) and different datasets. Additionally, NGL was applied to the large-scale dataset ImageNet-1k and segmentation tasks (Pascal VOC 2012 and COCO-Stuff164k datasets), further demonstrating its effectiveness and generalization ability.
In summary, this paper successfully developed a new loss function, NGL, through a Genetic Programming approach. This function not only excels in image classification tasks but also shows good performance in segmentation tasks, thereby addressing the key issues mentioned above.