Abstract:Ensemble learning is a methodology that integrates multiple DNN learners for improving prediction performance of individual learners. Diversity is greater when the errors of the ensemble prediction is more uniformly distributed. Greater diversity is highly correlated with the increase in ensemble accuracy. Another attractive property of diversity optimized ensemble learning is its robustness against deception: an adversarial perturbation attack can mislead one DNN model to misclassify but may not fool other ensemble DNN members consistently. In this paper we first give an overview of the concept of ensemble diversity and examine the three types of ensemble diversity in the context of DNN classifiers. We then describe a set of ensemble diversity measures, a suite of algorithms for creating diversity ensembles and for performing ensemble consensus (voted or learned) for generating high accuracy ensemble output by strategically combining outputs of individual members. This paper concludes with a discussion on a set of open issues in quantifying ensemble diversity for robust deep learning.

What problem does this paper attempt to address?

The main problem that this paper attempts to solve is to improve the diversity and robustness of deep neural network (DNN) ensemble learning, so as to enhance the model's defense ability against adversarial attacks. Specifically, the paper focuses on the following aspects: 1. **Improving prediction performance**: Improve the overall prediction performance by integrating multiple DNN models and reduce the error of a single model. 2. **Increasing diversity**: Explore how to create DNN ensembles with high diversity, making different models in the ensemble more evenly distributed in error, thereby improving the overall accuracy. 3. **Enhancing robustness**: Research how to make DNN ensembles more robust against deceptive attacks (such as adversarial examples), ensuring that even if some member models are misled, the entire ensemble can still maintain high prediction accuracy. ### Core contributions of the paper - **Defining and classifying diversity**: For the first time, the paper defines ensemble diversity from three perspectives: - **Type 1 diversity**: By changing the training data set, initial weights, network structure or hyper - parameters, etc., there are structural differences between different DNN models. - **Type 2 diversity**: By evaluating the output differences between different models (such as failure independence), promote the error - uncorrelatedness between member models. - **Type 3 diversity**: By changing the way of traversing the hypothesis space during the training process, make different classifiers converge to different hypotheses. - **Proposing diversity measurement indicators**: In order to quantify different types of diversity, a variety of measurement indicators are proposed, such as Q - statistic, correlation coefficient, binary divergence measure and Kappa - statistic, etc. - **Developing ensemble creation algorithms**: Design a set of algorithms for creating DNN ensembles with high diversity, and verify the performance of these ensembles under adversarial attacks through experiments. - **Exploring consensus methods**: Research how to combine the outputs of member models to generate highly accurate ensemble prediction results, especially in the face of adversarial examples. ### Experimental verification The paper evaluates the defense effect of the proposed diversity ensemble method under 12 different attacks (including non - targeted and targeted attacks) by conducting experiments on multiple benchmark data sets such as MNIST, CIFAR - 10 and ImageNet. The results show that the ensemble model with high diversity exhibits stronger robustness in adversarial attacks. ### Conclusion Through theoretical analysis and experimental verification, the paper proves that optimizing ensemble diversity can significantly improve the prediction performance of DNN models and the robustness against adversarial attacks. This provides new ideas and methods for constructing more powerful deep - learning systems. --- Hope this summary can help you understand the core problems of this paper and their solutions. If you have more questions or need further explanation, please feel free to let me know!

Deep Neural Network Ensembles against Deception: Ensemble Diversity, Accuracy and Robustness

Neural Network Ensembles: Theory, Training, and the Importance of Explicit Diversity

Efficient Diversity-Driven Ensemble for Deep Neural Networks

Improving Adversarial Robustness via Promoting Ensemble Diversity.

Improving Adversarial Robustness Via Promoting Ensemble Diversity

DICE: Diversity in Deep Ensembles via Conditional Redundancy Adversarial Estimation

“In-Network Ensemble”: Deep Ensemble Learning with Diversified Knowledge Distillation

Pathologies of Predictive Diversity in Deep Ensembles

Diversity supporting robustness: Enhancing adversarial robustness via differentiated ensemble predictions

Negatively correlated ensemble against transfer adversarial attacks

The Diversified Ensemble Neural Network.

Joint Training of Deep Ensembles Fails Due to Learner Collusion

Deep Ensembles Work, But Are They Necessary?

A DNN-Ensemble Method for Error Reduction and Training Data Selection in DNN Based Modeling

Exploring Model Learning Heterogeneity for Boosting Ensemble Robustness

Dynamic ensemble selection based on Deep Neural Network Uncertainty Estimation for Adversarial Robustness

Deep Ensembles: A Loss Landscape Perspective

The Disparate Benefits of Deep Ensembles

Developing parsimonious ensembles using ensemble diversity within a reinforcement learning framework

Revisiting Ensembles in an Adversarial Context: Improving Natural Accuracy

Ensemble Adversarial Defense via Integration of Multiple Dispersed Low Curvature Models