Ensembling is one approach that improves the performance of a neural network
by combining a number of independent neural networks, usually by either
averaging or summing up their individual outputs. We modify this ensembling
approach by training the sub-networks concurrently instead of independently.
This concurrent training of sub-networks leads them to cooperate with each
other, and we refer to them as "cooperative ensemble". Meanwhile, the
mixture-of-experts approach improves a neural network performance by dividing
up a given dataset to its sub-networks. It then uses a gating network that
assigns a specialization to each of its sub-networks called "experts". We
improve on these aforementioned ways for combining a group of neural networks
by using a k-Winners-Take-All (kWTA) activation function, that acts as the
combination method for the outputs of each sub-network in the ensemble. We
refer to this proposed model as "kWTA ensemble neural networks" (kWTA-ENN).
With the kWTA activation function, the losing neurons of the sub-networks are
inhibited while the winning neurons are retained. This results in sub-networks
having some form of specialization but also sharing knowledge with one another.
We compare our approach with the cooperative ensemble and mixture-of-experts,
where we used a feed-forward neural network with one hidden layer having 100
neurons as the sub-network architecture. Our approach yields a better
performance compared to the baseline models, reaching the following test
accuracies on benchmark datasets: 98.34% on MNIST, 88.06% on Fashion-MNIST,
91.56% on KMNIST, and 95.97% on WDBC.

基于 k 取胜神经元激活函数的合作集成神经网络（kWTA-ENN）是一种改进的集成方法，通过并行训练子网络并利用 k 取胜神经元激活函数，实现了专业化与知识共享的平衡，其在基准数据集上达到更好性能，例如 98.34％的 MNIST 测试准确率。