Abstract:Ensembling is one approach that improves the performance of a neural network by combining a number of independent neural networks, usually by either averaging or summing up their individual outputs. We modify this ensembling approach by training the sub-networks concurrently instead of independently. This concurrent training of sub-networks leads them to cooperate with each other, and we refer to them as "cooperative ensemble". Meanwhile, the mixture-of-experts approach improves a neural network performance by dividing up a given dataset to its sub-networks. It then uses a gating network that assigns a specialization to each of its sub-networks called "experts". We improve on these aforementioned ways for combining a group of neural networks by using a k-Winners-Take-All (kWTA) activation function, that acts as the combination method for the outputs of each sub-network in the ensemble. We refer to this proposed model as "kWTA ensemble neural networks" (kWTA-ENN). With the kWTA activation function, the losing neurons of the sub-networks are inhibited while the winning neurons are retained. This results in sub-networks having some form of specialization but also sharing knowledge with one another. We compare our approach with the cooperative ensemble and mixture-of-experts, where we used a feed-forward neural network with one hidden layer having 100 neurons as the sub-network architecture. Our approach yields a better performance compared to the baseline models, reaching the following test accuracies on benchmark datasets: 98.34% on MNIST, 88.06% on Fashion-MNIST, 91.56% on KMNIST, and 95.97% on WDBC.

Hybrid Ensemble Construction with Selected Neural Networks.

Optimizing Weights by Genetic Algorithm for Neural Network Ensemble

Artificial Neural Network Ensemble For Land Cover Classification

A Constructive Algorithm for Training Neural Network Ensemble

Multiobjective Neural Network Ensembles Based on Regularized Negative Correlation Learning.

A hybrid ensemble method with negative correlation learning for regression

Genetic Algorithm based Selective Neural Network Ensemble.

A Study of Ensemble of Hybrid Networks with Strong Regularization

Neural Network Ensemble With Evolutionary Algorithm for Highly Imbalanced Classification

Compact Ensemble Model of Neural Networks

Ensembling Neural Networks: Many Could Be Better Than All

Regularized Negative Correlation Learning for Neural Network Ensembles

Neural Subnetwork Ensembles

Improving Deep Neural Network Ensembles Using Reconstruction Error

Evolutionary Random Neural Ensembles Based on Negative Correlation Learning

Research and development of neural network ensembles: a survey

Boosting and Other Ensemble Methods

k-Winners-Take-All Ensemble Neural Network

Fast Decorrelated Neural Network Ensembles with Random Weights

COMBINING REGRESSION ESTIMATORS: GA-BASED SELECTIVE NEURAL NETWORK ENSEMBLE

The Diversified Ensemble Neural Network.