We investigate the problem of determining the predictive confidence (or, conversely, uncertainty) of a neural classifier through the lens of low-resource languages. By training models on sub-sampled datasets in three different languages, we assess the quality of estimates from a wide array of approaches and their dependence on the amount of available data. We find that while approaches based on pre-trained models and ensembles achieve the best results overall, the quality of uncertainty estimates can surprisingly suffer with more data. We also perform a qualitative analysis of uncertainties on sequences, discovering that a model's total uncertainty seems to be influenced to a large degree by its data uncertainty, not model uncertainty. All model implementations are open-sourced in a software package.

研究在从低资源语言中采样的数据集上训练模型的情况下，通过多种方法评估和分析神经分类器的预测置信度，发现尽管使用预训练模型和集成模型可以获得最佳结果，但数据集规模增大时不确定性估计的质量可能会受到影响。对序列不确定性进行了定性分析，发现模型的总不确定性在很大程度上受到数据不确定性的影响，并提供了开源的软件包。

探索自然语言处理中的预测不确定性和校准: 对方法和数据稀缺性影响的研究