Abstract:This paper presents a comprehensive empirical investigation into the interactions between various randomization techniques in Deep Neural Networks (DNNs) and their impact on learning performance. It is well-established that injecting randomness into the training process of DNNs, through various approaches, at different stages, is often beneficial for reducing overfitting and improving generalization. Nonetheless, the interactions between randomness techniques such as weight noise, dropout, and many others remain poorly understood. Consequently, it is challenging to determine which methods can be effectively combined to optimize DNN performance. To address this issue, we categorize the existing randomness techniques into four key types: injection of noise/randomness at the data, model structure, optimization or learning stage. We use this classification to identify gaps in the current coverage of potential mechanisms for the introduction of randomness, leading to proposing two new techniques: adding noise to the loss function and random masking of the gradient updates. In our empirical study, we employ a Particle Swarm Optimizer (PSO) for hyperparameter optimization (HPO) to explore the space of possible configurations to determine where and how much randomness should be injected to maximize DNN performance. We assess the impact of various types and levels of randomness for DNN architectures across standard computer vision benchmarks: MNIST, FASHION-MNIST, CIFAR10, and CIFAR100. Across more than 30 000 evaluated configurations, we perform a detailed examination of the interactions between randomness techniques and their combined impact on DNN performance. Our findings reveal that randomness through data augmentation and in weight initialization are the main contributors to performance improvement. Additionally, correlation analysis demonstrates that different optimizers, such as Adam and Gradient Descent with Momentum, prefer distinct types of randomization during the training process. A GitHub repository with the complete implementation and generated dataset is available 1 .

An Examination of On-Line Machine Learning Approaches for Pseudo-Random Generated Data

Genetic Ensemble of Extreme Learning Machine

Pseudo-random Number Generator Influences on Average Treatment Effect Estimates Obtained with Machine Learning

GAN-based Pseudo Random Number Generation Optimized Through Genetic Algorithms

A Search for Good Pseudo-random Number Generators : Survey and Empirical Studies

Random Bits Regression: a Strong General Predictor for Big Data

Rolling the Dice for Better Deep Learning Performance: A Study of Randomness Techniques in Deep Neural Networks

Learning Performance of Weighted Distributed Learning With Support Vector Machines

Benchmarking a Quantum Random Number Generator with Machine Learning

Randomness in Neural Networks: an Overview

Cumulative Probability Distribution Model for Evaluating User Behavior Prediction Algorithms

Quantifying Inherent Randomness in Machine Learning Algorithms

Analysis of Logistic Map for Pseudorandom Number Generation in Game Development

Reproducibility, energy efficiency and performance of pseudorandom number generators in machine learning: a comparative study of python, numpy, tensorflow, and pytorch implementations

Learning Predictions for Algorithms with Predictions

Machine Learning Predictors for Min-Entropy Estimation

Randomized Prediction Games for Adversarial Machine Learning

A dynamic self-adoptive genetic algorithm for personal credit risk assessment

Learning from Pseudo-Randomness With an Artificial Neural Network - Does God Play Pseudo-Dice?

RSG: A Simple but Effective Module for Learning Imbalanced Datasets

Machine Learning and Sampling Scheme: An Empirical Study of Money Laundering Detection