We present an analysis of the performance of machine learning classifiers on
discriminating between similar languages and language varieties. We carried out
a number of experiments using the results of the two editions of the
Discriminating between Similar Languages (DSL) shared task. We investigate the
progress made between the two tasks, estimate an upper bound on possible
performance using ensemble and oracle combination, and provide learning curves
to help us understand which languages are more challenging. A number of
difficult sentences are identified and investigated further with human
annotation.

本文分析了机器学习分类器在区分相似语言和语言变体方面的表现，使用 DSL 共享任务的两个版本的结果进行了一系列实验，研究了两个任务之间的进展，估计了可能性能的上限，并提供了学习曲线，以帮助我们了解哪些语言更具挑战性。此外，还使用人工注释对一些困难的句子进行了进一步的研究。

辨别相似语言：评估与探索

Discriminating Similar Languages: Evaluations and Explorations

In biospectroscopy, suitably annotated and statistically independent samples
(e. g. patients, batches, etc.) for classifier training and testing are scarce
and costly. Learning curves show the model performance as function of the
training sample size and can help to determine the sample size needed to train
good classifiers. However, building a good model is actually not enough: the
performance must also be proven. We discuss learning curves for typical small
sample size situations with 5 - 25 independent samples per class. Although the
classification models achieve acceptable performance, the learning curve can be
completely masked by the random testing uncertainty due to the equally limited
test sample size. In consequence, we determine test sample sizes necessary to
achieve reasonable precision in the validation and find that 75 - 100 samples
will usually be needed to test a good but not perfect classifier. Such a data
set will then allow refined sample size planning on the basis of the achieved
performance. We also demonstrate how to calculate necessary sample sizes in
order to show the superiority of one classifier over another: this often
requires hundreds of statistically independent test samples or is even
theoretically impossible. We demonstrate our findings with a data set of ca.
2550 Raman spectra of single cells (five classes: erythrocytes, leukocytes and
three tumour cell lines BT-20, MCF-7 and OCI-AML3) as well as by an extensive
simulation that allows precise determination of the actual performance of the
models in question.

本研究讨论了常见的小样本分类问题，通过绘制学习曲线计算训练样本集大小，确定测试样本集合适的大小，以验证分类器的性能，并使用单细胞的光谱数据集和大规模模拟来演示我们的研究结果。

分类模型的样本量规划

Sample Size Planning for Classification Models

Learning curves for Gaussian process regression are well understood when the
`student' model happens to match the `teacher' (true data generation process).
I derive approximations to the learning curves for the more generic case of
mismatched models, and find very rich behaviour: For large input space
dimensionality, where the results become exact, there are universal
(student-independent) plateaux in the learning curve, with transitions in
between that can exhibit arbitrarily many over-fitting maxima. In lower
dimensions, plateaux also appear, and the asymptotic decay of the learning
curve becomes strongly student-dependent. All predictions are confirmed by
simulations.

通过对于不匹配的模型进行学习曲线的近似，对于高维情况学生无关的平原在学习曲线中出现，而在低维情况下，学习曲线的渐近衰减强烈依赖于学生。

不匹配模型下的高斯过程回归

Gaussian Process Regression with Mismatched Models

We consider the problem of calculating learning curves (i.e., average
generalization performance) of Gaussian processes used for regression. On the
basis of a simple expression for the generalization error, in terms of the
eigenvalue decomposition of the covariance function, we derive a number of
approximation schemes. We identify where these become exact, and compare with
existing bounds on learning curves; the new approximations, which can be used
for any input space dimension, generally get substantially closer to the truth.
We also study possible improvements to our approximations. Finally, we use a
simple exactly solvable learning scenario to show that there are limits of
principle on the quality of approximations and bounds expressible solely in
terms of the eigenvalue spectrum of the covariance function.

该论文考虑了使用高斯过程进行回归时，计算学习曲线（即平均泛化性能）的问题，并提出了一种基于协方差函数的特征值分解的一些近似方案，这些新的近似方案通常更接近真实值。