This paper presents our contribution to the PolEval 2021 Task 2: Evaluation
of translation quality assessment metrics. We describe experiments with
pre-trained language models and state-of-the-art frameworks for translation
quality assessment in both nonblind and blind versions of the task. Our
solutions ranked second in the nonblind version and third in the blind version.

本文介绍了我们在 PolEval 2021 任务 2 中对翻译质量评估指标进行评估的贡献，我们通过预训练语言模型和最先进的框架在非盲和盲版本的任务中进行了实验，我们的解决方案在非盲版本排名第二，在盲版本排名第三。

基于神经网络的英波机器翻译质量评估方法研究

Approaching English-Polish Machine Translation Quality Assessment with Neural-based Methods

Translation Quality Assessment (TQA) is a process conducted by human
translators and is widely used, both for estimating the performance of
(increasingly used) Machine Translation, and for finding an agreement between
translation providers and their customers. While translation scholars are aware
of the importance of having a reliable way to conduct the TQA process, it seems
that there is limited literature that tackles the issue of reliability with a
quantitative approach. In this work, we consider the TQA as a complex process
from the point of view of physics of complex systems and approach the
reliability issue from the Bayesian paradigm. Using a dataset of translation
quality evaluations (in the form of error annotations), produced entirely by
the Professional Translation Service Provider Translated SRL, we compare two
Bayesian models that parameterise the following features involved in the TQA
process: the translation difficulty, the characteristics of the translators
involved in producing the translation, and of those assessing its quality - the
reviewers. We validate the models in an unsupervised setting and show that it
is possible to get meaningful insights into translators even with just one
review per translation; subsequently, we extract information like translators'
skills and reviewers' strictness, as well as their consistency in their
respective roles. Using this, we show that the reliability of reviewers cannot
be taken for granted even in the case of expert translators: a translator's
expertise can induce a cognitive bias when reviewing a translation produced by
another translator. The most expert translators, however, are characterised by
the highest level of consistency, both in translating and in assessing the
translation quality.

本文通过基于复杂系统物理和贝叶斯范畴的方法，考虑了机器翻译的翻译质量评估问题，并比较了两种不同的参数化模型。结果表明，即使仅仅进行一次评估，也可以从中获得有意义的信息，如译者技能和评估者的严格度，以及他们在各自角色中的一致性。此外，研究表明，即使是专家译者也可能会出现认知偏差，因此评估者的可靠性也并不能保证，但最专业的翻译者具有最高水平的一致性。

翻译者可靠性评估的贝叶斯方法

A Bayesian approach to translators' reliability assessment

The study of the applicability of the BERTScore metric was conducted to
translation quality assessment at the sentence level for English -> Russian
direction. Experiments were performed with a pre-trained Multilingual BERT as
well as with a pair of Monolingual BERT models. To align monolingual
embeddings, an orthogonal transformation based on anchor tokens was used. It
was demonstrated that such transformation helps to prevent mismatching issue
and shown that this approach gives better results than using embeddings of the
Multilingual model. To improve the token matching process it is proposed to
combine all incomplete WorkPiece tokens into meaningful words and use simple
averaging of corresponding vectors and to calculate BERTScore based on anchor
tokens only. Such modifications allowed us to achieve a better correlation of
the model predictions with human judgments. In addition to evaluating machine
translation, several versions of human translation were evaluated as well, the
problems of this approach were listed.

该研究旨在利用 BERTScore 度量英俄翻译中句子级别的翻译质量，并通过利用单语 BERT 模型和基于锚定标记的正交变换对齐单语言嵌入来改进模型性能，结果表明改进的方法可以提高模型预测与人类判断的相关性。