Independently trained machine learning models tend to learn similar features.
Given an ensemble of independently trained models, this results in correlated
predictions and common failure modes. Previous attempts focusing on
decorrelation of output predictions or logits yielded mixed results,
particularly due to their reduction in model accuracy caused by conflicting
optimization objectives. In this paper, we propose the novel idea of utilizing
methods of the representational similarity field to promote dissimilarity
during training instead of measuring similarity of trained models. To this end,
we promote intermediate representations to be dissimilar at different depths
between architectures, with the goal of learning robust ensembles with disjoint
failure modes. We show that highly dissimilar intermediate representations
result in less correlated output predictions and slightly lower error
consistency, resulting in higher ensemble accuracy. With this, we shine first
light on the connection between intermediate representations and their impact
on the output predictions.

利用表征相似性领域的方法，提出了一种新颖的想法，在训练期间推动中间表示在不同架构之间的深度上不同，以学习具有不同故障模式的鲁棒集成模型。结果表明，高度不相似的中间表示导致较低的相关性输出预测及稍微降低的错误一致性，并提高了集成模型的准确性。