With the recent advance in neural machine translation demonstrating its importance, research on quality estimation (QE) has been steadily progressing. QE aims to automatically predict the quality of machine translation (MT) output without reference sentences. Despite its high utility in the real world, there remain several limitations concerning manual QE data creation: inevitably incurred non-trivial costs due to the need for translation experts, and issues with data scaling and language expansion. To tackle these limitations, we present QUAK, a Korean-English synthetic QE dataset generated in a fully automatic manner. This consists of three sub-QUAK datasets QUAK-M, QUAK-P, and QUAK-H, produced through three strategies that are relatively free from language constraints. Since each strategy requires no human effort, which facilitates scalability, we scale our data up to 1.58M for QUAK-P, H and 6.58M for QUAK-M. As an experiment, we quantitatively analyze word-level QE results in various ways while performing statistical analysis. Moreover, we show that datasets scaled in an efficient way also contribute to performance improvements by observing meaningful performance gains in QUAK-M, P when adding data up to 1.58M.

本文提出QUAK, 是一个自动生成的合成QE数据集（包括三个子数据集），可用于提高机器翻译输出质量的自动预测。该数据集是通过三种相对自由的生成策略产生的，具有良好的可扩展性，已扩展到1.58M到6.58M。对数据集进行统计分析后发现，QUAK-M，P中添加数据可获得显著性能提升。

QUAK：一份用于韩英神经机器翻译的合成质量估计数据集