Mutation validation (MV) is a recently proposed approach for model selection,
garnering significant interest due to its unique characteristics and potential
benefits compared to the widely used cross-validation (CV) method. In this
study, we empirically compared MV and $k$-fold CV using benchmark and
real-world datasets. By employing Bayesian tests, we compared generalization
estimates yielding three posterior probabilities: practical equivalence, CV
superiority, and MV superiority. We also evaluated the differences in the
capacity of the selected models and computational efficiency. We found that
both MV and CV select models with practically equivalent generalization
performance across various machine learning algorithms and the majority of
benchmark datasets. MV exhibited advantages in terms of selecting simpler
models and lower computational costs. However, in some cases MV selected overly
simplistic models leading to underfitting and showed instability in
hyperparameter selection. These limitations of MV became more evident in the
evaluation of a real-world neuroscientific task of predicting sex at birth
using brain functional connectivity.

通过比较变异验证方法和 k 折交叉验证方法，我们发现两种方法在选择模型的泛化性能上表现相当，但变异验证方法在选择简化模型和计算成本方面具有优势。然而，在某些情况下，变异验证方法选择过于简化的模型，导致欠拟合，并在超参数选择方面表现不稳定。