Almost all frameworks for the manual or automatic evaluation of machine
translation characterize the quality of an MT output with a single number. An
exception is the Multidimensional Quality Metrics (MQM) framework which offers
a fine-grained ontology of quality dimensions for scoring (such as style,
fluency, accuracy, and terminology). Previous studies have demonstrated the
feasibility of MQM annotation but there are, to our knowledge, no computational
models that predict MQM scores for novel texts, due to a lack of resources. In
this paper, we address these shortcomings by (a) providing a 1200-sentence MQM
evaluation benchmark for the language pair English-Korean and (b) reframing MT
evaluation as the multi-task problem of simultaneously predicting several MQM
scores using SOTA language models, both in a reference-based MT evaluation
setup and a reference-free quality estimation (QE) setup. We find that
reference-free setup outperforms its counterpart in the style dimension while
reference-based models retain an edge regarding accuracy. Overall, RemBERT
emerges as the most promising model. Through our evaluation, we offer an
insight into the translation quality in a more fine-grained, interpretable
manner.

通过提供一个英韩语言对的 1200 句 MQM 评估基准，将机器翻译评估重新定义为使用 SOTA 语言模型同时预测多个 MQM 分数的多任务问题，在参考 MT 评估和无参考质量估计（QE）设置中，我们发现无参考设置在风格维度上优于参考设置，而参考模型在准确性方面保持优势，总体上，RemBERT 是最有希望的模型。通过我们的评估，以一种更精细化、可解释的方式提供了关于翻译质量的见解。

多维机器翻译评估：韩语模型评估和资源

Multi-Dimensional Machine Translation Evaluation: Model Evaluation and  Resource for Korean

Making an informed choice of pre-trained language model (LM) is critical for
performance, yet environmentally costly, and as such widely underexplored. The
field of Computer Vision has begun to tackle encoder ranking, with promising
forays into Natural Language Processing, however they lack coverage of
linguistic tasks such as structured prediction. We propose probing to rank LMs,
specifically for parsing dependencies in a given language, by measuring the
degree to which labeled trees are recoverable from an LM's contextualized
embeddings. Across 46 typologically and architecturally diverse LM-language
pairs, our probing approach predicts the best LM choice 79% of the time using
orders of magnitude less compute than training a full parser. Within this
study, we identify and analyze one recently proposed decoupled LM - RemBERT -
and find it strikingly contains less inherent dependency information, but often
yields the best parser after full fine-tuning. Without this outlier our
approach identifies the best LM in 89% of cases.

通过衡量语言模型上下文嵌入中恢复标记树的程度，我们提出使用考察探测来对语言模型进行排序，从而确定适合特定语言的最佳模型选择。在 46 种不同类型和结构的语言模型 - 语言对中，我们的探测方法预测最佳语言模型选择的准确率为 79％，比训练完整解析器需要更少的计算量。在本研究中，我们发现 RemBERT 是一个最近提出的解耦合语言模型，它显著包含较少的固有依赖信息，但通常在完全微调后可以产生最佳的解析器结果。在排除这个异常值后，我们的方法在 89％的情况下确定最佳的语言模型选择。

基于结构的排序：依赖探测的语言模型排名

Sort by Structure: Language Model Ranking as Dependency Probing

Skill Classification (SC) is the task of classifying job competences from job
postings. This work is the first in SC applied to Danish job vacancy data. We
release the first Danish job posting dataset: Kompetencer (en: competences),
annotated for nested spans of competences. To improve upon coarse-grained
annotations, we make use of The European Skills, Competences, Qualifications
and Occupations (ESCO; le Vrang et al., 2014) taxonomy API to obtain
fine-grained labels via distant supervision. We study two setups: The zero-shot
and few-shot classification setting. We fine-tune English-based models and
RemBERT (Chung et al., 2020) and compare them to in-language Danish models. Our
results show RemBERT significantly outperforms all other models in both the
zero-shot and the few-shot setting.

本研究使用 ESCO 分类法 API 对丹麦职位招聘数据集 Kompetencer 进行标注并比较数据集的 fine-grained 标签应用于 zero-shot 和 few-shot 分类设置，结果表明 RemBERT 在两种情况下的表现均优于其他模型。