Quantification is the supervised learning task that consists of training
predictors of the class prevalence values of sets of unlabelled data, and is of
special interest when the labelled data on which the predictor has been trained
and the unlabelled data are not IID, i.e., suffer from dataset shift. To date,
quantification methods have mostly been tested only on a special case of
dataset shift, i.e., prior probability shift; the relationship between
quantification and other types of dataset shift remains, by and large,
unexplored. In this work we carry out an experimental analysis of how current
quantification algorithms behave under different types of dataset shift, in
order to identify limitations of current approaches and hopefully pave the way
for the development of more broadly applicable methods. We do this by proposing
a fine-grained taxonomy of types of dataset shift, by establishing protocols
for the generation of datasets affected by these types of shift, and by testing
existing quantification methods on the datasets thus generated. One finding
that results from this investigation is that many existing quantification
methods that had been found robust to prior probability shift are not
necessarily robust to other types of dataset shift. A second finding is that no
existing quantification method seems to be robust enough to dealing with all
the types of dataset shift we simulate in our experiments. The code needed to
reproduce all our experiments is publicly available at
this https URL

在这项研究中，我们通过实验分析了当前的定量化算法在不同类型的数据集偏移下的行为，以识别现有方法的局限性，并为开发更广泛适用的方法铺平道路。我们通过提出细粒度的数据集偏移类型分类，建立受这些偏移类型影响的数据集生成协议，并在生成的数据集上测试现有的定量化方法。研究结果表明，许多已被发现对于先验概率偏移具有鲁棒性的定量化方法对于其他类型的数据集偏移可能不具备鲁棒性。此外，没有发现任何现有的定量化方法能够对我们在实验中模拟的所有数据集偏移类型具备足够的鲁棒性。

二进制量化和数据集变化：实验调查

Binary Quantification and Dataset Shift: An Experimental Investigation

For the binary prevalence quantification problem under prior probability
shift, we determine the asymptotic variance of the maximum likelihood
estimator. We find that it is a function of the Brier score for the regression
of the class label on the features under the test data set distribution. This
observation suggests that optimising the accuracy of a base classifier, as
measured by the Brier score, on the training data set helps to reduce the
variance of the related quantifier on the test data set. Therefore, we also
point out training criteria for the base classifier that imply optimisation of
both of the Brier scores on the training and the test data sets.

通过研究二元预测问题、先验概率转移及基础分类器的优化策略，本文确定其最大似然估计量的渐近方差，发现其与特征下类别标签的 Brier 得分相关，并点明了训练基础分类器的方法以最优化训练集和测试集的 Brier 得分。