Characterizing the patterns of errors that a system makes helps researchers
focus future development on increasing its accuracy and robustness. We propose
a novel form of "meta learning" that automatically learns interpretable rules
that characterize the types of errors that a system makes, and demonstrate
these rules' ability to help understand and improve two NLP systems. Our
approach works by collecting error cases on validation data, extracting
meta-features describing these samples, and finally learning rules that
characterize errors using these features. We apply our approach to VilBERT, for
Visual Question Answering, and RoBERTa, for Common Sense Question Answering.
Our system learns interpretable rules that provide insights into systemic
errors these systems make on the given tasks. Using these insights, we are also
able to "close the loop" and modestly improve performance of these systems.

本研究提出了一种自动学习可解释规则的元学习方法，应用在 NLP 系统上，用以了解和改善系统的错误表现，并取得了一定的性能提升。该方法通过对验证数据的错误案例进行元特征提取以及规则学习，成功地为 VilBERT 和 RoBERTa 建立了可解释规则，进一步提高了系统表现。

自动化错误分析：学习错误特征

Towards Automated Error Analysis: Learning to Characterize Errors

Customers of machine learning systems demand accountability from the
companies employing these algorithms for various prediction tasks.
Accountability requires understanding of system limit and condition of
erroneous predictions, as customers are often interested in understanding the
incorrect predictions, and model developers are absorbed in finding methods
that can be used to get incremental improvements to an existing system.
Therefore, we propose an accountable error characterization method, AEC, to
understand when and where errors occur within the existing black-box models.
AEC, as constructed with human-understandable linguistic features, allows the
model developers to automatically identify the main sources of errors for a
given classification system. It can also be used to sample for the set of most
informative input points for a next round of training. We perform error
detection for a sentiment analysis task using AEC as a case study. Our results
on the sample sentiment task show that AEC is able to characterize erroneous
predictions into human understandable categories and also achieves promising
results on selecting erroneous samples when compared with the uncertainty-based
sampling.

提出一种可追溯的错误特征化方法 (AEC)，用于理解现有黑盒模型中的错误发生时间和位置，并能自动识别分类系统中的错误源。以情感分析任务为案例研究，结果表明 AEC 能将错误预测归纳为易懂的类别并在选择错误样本方面实现了可靠的结果。