Trained models are often composed with post-hoc transforms such as
temperature scaling (TS), ensembling and stochastic weight averaging (SWA) to
improve performance, robustness, uncertainty estimation, etc. However, such
transforms are typically applied only after the base models have already been
finalized by standard means. In this paper, we challenge this practice with an
extensive empirical study. In particular, we demonstrate a phenomenon that we
call post-hoc reversal, where performance trends are reversed after applying
these post-hoc transforms. This phenomenon is especially prominent in
high-noise settings. For example, while base models overfit badly early in
training, both conventional ensembling and SWA favor base models trained for
more epochs. Post-hoc reversal can also suppress the appearance of double
descent and mitigate mismatches between test loss and test error seen in base
models. Based on our findings, we propose post-hoc selection, a simple
technique whereby post-hoc metrics inform model development decisions such as
early stopping, checkpointing, and broader hyperparameter choices. Our
experimental analyses span real-world vision, language, tabular and graph
datasets from domains like satellite imaging, language modeling, census
prediction and social network analysis. On an LLM instruction tuning dataset,
post-hoc selection results in > 1.5x MMLU improvement compared to naive
selection. Code is available at
this https URL

在这篇文章中，作者提出了后企业行的选择技术，通过后企业行指标在模型开发决策中，如早停，检查点和更广的超参数选择，对模型进行指导。