Assessing the performance of interpreting services is a complex task, given
the nuanced nature of spoken language translation, the strategies that
interpreters apply, and the diverse expectations of users. The complexity of
this task become even more pronounced when automated evaluation methods are
applied. This is particularly true because interpreted texts exhibit less
linearity between the source and target languages due to the strategies
employed by the interpreter.
This study aims to assess the reliability of automatic metrics in evaluating
simultaneous interpretations by analyzing their correlation with human
evaluations. We focus on a particular feature of interpretation quality, namely
translation accuracy or faithfulness. As a benchmark we use human assessments
performed by language experts, and evaluate how well sentence embeddings and
Large Language Models correlate with them. We quantify semantic similarity
between the source and translated texts without relying on a reference
translation. The results suggest GPT models, particularly GPT-3.5 with direct
prompting, demonstrate the strongest correlation with human judgment in terms
of semantic similarity between source and target texts, even when evaluating
short textual segments. Additionally, the study reveals that the size of the
context window has a notable impact on this correlation.

评估口译服务的表现是一项复杂的任务，尤其是在应用自动评估方法时，本研究旨在通过分析自动度量与人工评估之间的相关性来评估同传口译的可靠性，结果表明 GPT 模型，特别是 GPT-3.5 具有最强的语义相似性相关性，即使在评估短文本片段时也是如此。