Annotated datasets are an essential ingredient to train, evaluate, compare
and productionalize supervised machine learning models. It is therefore
imperative that annotations are of high quality. For their creation, good
quality management and thereby reliable quality estimates are needed. Then, if
quality is insufficient during the annotation process, rectifying measures can
be taken to improve it. Quality estimation is often performed by having experts
manually label instances as correct or incorrect. But checking all annotated
instances tends to be expensive. Therefore, in practice, usually only subsets
are inspected; sizes are chosen mostly without justification or regard to
statistical power and more often than not, are relatively small. Basing
estimates on small sample sizes, however, can lead to imprecise values for the
error rate. Using unnecessarily large sample sizes costs money that could be
better spent, for instance on more annotations. Therefore, we first describe in
detail how to use confidence intervals for finding the minimal sample size
needed to estimate the annotation error rate. Then, we propose applying
acceptance sampling as an alternative to error rate estimation We show that
acceptance sampling can reduce the required sample sizes up to 50% while
providing the same statistical guarantees.

使用置信区间寻找估计注释错误率所需的最小样本量，然后提出应用接受抽样作为错误率估计的一种替代方法，展示接受抽样可以减少最多 50% 的所需样本量，同时提供相同的统计保证。

关于数据标注的高效和统计质量估计

On Efficient and Statistical Quality Estimation for Data Annotation

In the context of industrially mass-manufactured products, quality management
is based on physically inspecting a small sample from a large batch and
reasoning about the batch's quality conformance. When complementing physical
inspections with predictions from machine learning models, it is crucial that
the uncertainty of the prediction is known. Otherwise, the application of
established quality management concepts is not legitimate. Deterministic
(machine learning) models lack quantification of their predictive uncertainty
and are therefore unsuitable. Probabilistic (machine learning) models provide a
predictive uncertainty along with the prediction. However, a concise
relationship is missing between the measurement uncertainty of physical
inspections and the predictive uncertainty of probabilistic models in their
application in quality management. Here, we show how the predictive uncertainty
of probabilistic (machine learning) models is related to the measurement
uncertainty of physical inspections. This enables the use of probabilistic
models for virtual inspections and integrates them into existing quality
management concepts. Thus, we can provide a virtual measurement for any quality
characteristic based on the process data and achieve a 100 percent inspection
rate. In the field of Predictive Quality, the virtual measurement is of great
interest. Based on our results, physical inspections with a low sampling rate
can be accompanied by virtual measurements that allow an inspection rate of 100
percent. We add substantial value, especially to complex process chains, as
faulty products/parts are identified promptly and upcoming process steps can be
aborted.

通过将机器学习模型的预测不确定性与实际物理检测的测量不确定性联系起来，可以在质量管理中使用概率模型进行虚拟检测，并将其融入现有质量管理概念，从而实现对过程数据的全面测量，提高检测率并及时识别缺陷产品 / 零件。

测量不确定性：关联物理和虚拟测量的不确定性

Measurement Uncertainty: Relating the uncertainties of physical and  virtual measurements

Quality management and assurance is key for space agencies to guarantee the
success of space missions, which are high-risk and extremely costly. In this
paper, we present a system to generate quizzes, a common resource to evaluate
the effectiveness of training sessions, from documents about quality assurance
procedures in the Space domain. Our system leverages state of the art
auto-regressive models like T5 and BART to generate questions, and a RoBERTa
model to extract answers for such questions, thus verifying their suitability.

本文提出了一个系统，可以从关于太空质量保证程序的文档中生成测验，以评估培训效果。该系统利用了 T5 和 BART 等最先进的自回归模型来生成问题，并使用 RoBERTa 模型提取答案，从而验证问题的适用性。

为支持空间科学与工程质量管理与保障培训生成测验

Generating Quizzes to Support Training on Quality Management and Assurance in Space Science and Engineering

In the past decade, Artificial Intelligence (AI) has become a part of our
daily lives due to major advances in Machine Learning (ML) techniques. In spite
of an explosive growth in the raw AI technology and in consumer facing
applications on the internet, its adoption in business applications has
conspicuously lagged behind. For business/mission-critical systems, serious
concerns about reliability and maintainability of AI applications remain. Due
to the statistical nature of the output, software 'defects' are not well
defined. Consequently, many traditional quality management techniques such as
program debugging, static code analysis, functional testing, etc. have to be
reevaluated. Beyond the correctness of an AI model, many other new quality
attributes, such as fairness, robustness, explainability, transparency, etc.
become important in delivering an AI system. The purpose of this paper is to
present a view of a holistic quality management framework for ML applications
based on the current advances and identify new areas of software engineering
research to achieve a more trustworthy AI.

本文提出了一个基于现有进展的全面质量管理框架视角，并确定了软件工程研究的新领域，以实现更可靠的人工智能。