Finding Strong Lottery Ticket Networks with Genetic Algorithms

Philipp Altmann,Julian Schönberger,Maximilian Zorn,Thomas Gabor

2024-11-07

Abstract:According to the Strong Lottery Ticket Hypothesis, every sufficiently large neural network with randomly initialized weights contains a sub-network which - still with its random weights - already performs as well for a given task as the trained super-network. We present the first approach based on a genetic algorithm to find such strong lottery ticket sub-networks without training or otherwise computing any gradient. We show that, for smaller instances of binary classification tasks, our evolutionary approach even produces smaller and better-performing lottery ticket networks than the state-of-the-art approach using gradient information.

Neural and Evolutionary Computing

What problem does this paper attempt to address?

The core problem that this paper attempts to solve is: **How to use the genetic algorithm to find strong lottery ticket sub - networks without training or calculating gradients**.

### Problem Background

According to the Strong Lottery Ticket Hypothesis (SLTH), in a large enough neural network with randomly initialized weights, there exists a sub - network that, while keeping its random weights, can already perform a given task as well as the trained super - network. However, finding such a sub - network usually requires a large amount of computing resources because the number of possible connection combinations grows exponentially with the network size.

### Main Contributions of the Paper

1. **Proposed a new method based on the genetic algorithm**:
- This method can find strong lottery ticket sub - networks without training weights or using gradient information.
- This is the first research attempt to find strong lottery ticket sub - networks only through combinatorial optimization (rather than gradient information).

2. **Demonstrated the effectiveness of the genetic algorithm**:
- In smaller binary classification task examples, the lottery sub - networks evolved by the genetic algorithm are smaller and perform better than the sub - networks produced by the state - of - the - art gradient - based methods.

3. **Explored the application potential of the genetic algorithm**:
- The researchers hope that this work will pave the way for further research on the application of the genetic algorithm in neural network optimization, especially when dealing with non - continuous functions and complex optimization landscapes.

### Method Overview

- **Solution Representation**: Use a binary mask to represent the selection of sub - networks, that is, whether each weight is retained.
- **Fitness Evaluation and Selection**: The main objective is to find the accuracy that matches the standard training, and the secondary objective is to reduce the number of parameters in the sub - network (sparsity) as much as possible. Fitness is evaluated based on the accuracy on the training data set.
- **Genetic Operations**: Include single - point mutation, random crossover, etc., to generate new individuals and increase the population diversity.

### Experimental Results

- **Verified on multiple data sets and network architectures**, and the results show that the genetic algorithm can find sub - networks with similar accuracy to gradient - descent - trained ones in some cases.
- **Especially on small networks**, the effect of the genetic algorithm is particularly significant and even exceeds the gradient - based methods.

In conclusion, this paper aims to provide a brand - new, training - free strong lottery sub - network search method through the genetic algorithm and demonstrates its potential in simplifying neural network structures.