Deep learning models have proven to be highly successful. Yet, their
over-parameterization gives rise to model multiplicity, a phenomenon in which
multiple models achieve similar performance but exhibit distinct underlying
behaviours. This multiplicity presents a significant challenge and necessitates
additional specifications in model selection to prevent unexpected failures
during deployment. While prior studies have examined these concerns, they focus
on individual metrics in isolation, making it difficult to obtain a
comprehensive view of multiplicity in trustworthy machine learning. Our work
stands out by offering a one-stop empirical benchmark of multiplicity across
various dimensions of model design and its impact on a diverse set of
trustworthy metrics. In this work, we establish a consistent language for
studying model multiplicity by translating several trustworthy metrics into
accuracy under appropriate interventions. We also develop a framework, which we
call multiplicity sheets, to benchmark multiplicity in various scenarios. We
demonstrate the advantages of our setup through a case study in image
classification and provide actionable insights into the impact and trends of
different hyperparameters on model multiplicity. Finally, we show that
multiplicity persists in deep learning models even after enforcing additional
specifications during model selection, highlighting the severity of
over-parameterization. The concerns of under-specification thus remain, and we
seek to promote a more comprehensive discussion of multiplicity in trustworthy
machine learning.

通过对可信度的多维度模型设计和其对一系列可靠度指标的影响进行一站式的经验基准测试，我们的研究暴露了深度学习模型的过参数化现象导致的模型多样性，这需要在模型选择过程中增加额外的规范以避免意外的失败，并为这种多样性提供影响和趋势的可行性建议。