Abstract:Quality-Diversity (QD) algorithms have emerged as a powerful optimization paradigm with the aim of generating a set of high-quality and diverse solutions. To achieve such a challenging goal, QD algorithms require maintaining a large archive and a large population in each iteration, which brings two main issues, sample and resource efficiency. Most advanced QD algorithms focus on improving the sample efficiency, while the resource efficiency is overlooked to some extent. Particularly, the resource overhead during the training process has not been touched yet, hindering the wider application of QD algorithms. In this paper, we highlight this important research question, i.e., how to efficiently train QD algorithms with limited resources, and propose a novel and effective method called RefQD to address it. RefQD decomposes a neural network into representation and decision parts, and shares the representation part with all decision parts in the archive to reduce the resource overhead. It also employs a series of strategies to address the mismatch issue between the old decision parts and the newly updated representation part. Experiments on different types of tasks from small to large resource consumption demonstrate the excellent performance of RefQD: it not only uses significantly fewer resources (e.g., 16\% GPU memories on QDax and 3.7\% on Atari) but also achieves comparable or better performance compared to sample-efficient QD algorithms. Our code is available at \url{<a class="link-external link-https" href="https://github.com/lamda-bbo/RefQD" rel="external noopener nofollow">this https URL</a>}.
What problem does this paper attempt to address?
The problem that this paper attempts to solve is: how to efficiently train Quality - Diversity (QD) algorithms under limited resource conditions. Specifically, although existing QD algorithms have made some improvements in sample efficiency, they still have deficiencies in resource efficiency. In particular, the storage and computing resources required during the training process are very large, which limits the wide use of QD algorithms in practical applications. Therefore, this paper proposes a new method, RefQD, aiming to improve the resource efficiency of QD algorithms by reducing resource overhead.
### Specific description of the problem
1. **Sample efficiency**: Existing QD algorithms have focused on how to reduce the number of samples required in the optimization process. However, these studies usually assume that computing resources are sufficient.
2. **Resource efficiency**: In fact, computing resources are usually limited, especially in practical application scenarios. QD algorithms need to maintain a large archive and a large number of solutions, which leads to significant storage and computing resource overhead. This high - resource requirement limits the application range of QD algorithms.
### Solutions
To address the above challenges, this paper proposes the RefQD method, whose main features include:
- **Network decomposition and sharing**: Decompose the neural network into a representation part and a decision part. The representation part is shared across the entire archive, while each solution only stores its decision part. This can significantly reduce resource overhead.
- **Overcoming the mismatch problem**: Due to the frequent update of the representation part, the decision part in the archive may become invalid. For this reason, RefQD introduces a series of strategies, such as regular re - evaluation, maintaining a Deep Decision Archive (DDA), top - k re - evaluation, and learning rate decay of the representation part, to ensure that the decision part can adapt to the latest representation part.
- **Experimental verification**: Through experiments on different types of tasks, it is proved that RefQD not only significantly reduces resource consumption (for example, the GPU memory usage is reduced by 16% to 3.7%), but also achieves performance comparable to or even better than existing methods.
### Main contributions
1. **For the first time, emphasized the resource efficiency problem in the training stage of QD algorithms**.
2. **Proposed RefQD, a novel and effective method, which improves the resource efficiency of QD algorithms**.
3. **Experimental results show that RefQD can still maintain high performance under limited resource conditions, and the resource consumption is significantly reduced**.
Through these improvements, RefQD enables QD algorithms to operate more effectively in resource - constrained environments, thereby expanding their potential in practical applications.