Making an informed choice of pre-trained language model (LM) is critical for
performance, yet environmentally costly, and as such widely underexplored. The
field of Computer Vision has begun to tackle encoder ranking, with promising
forays into Natural Language Processing, however they lack coverage of
linguistic tasks such as structured prediction. We propose probing to rank LMs,
specifically for parsing dependencies in a given language, by measuring the
degree to which labeled trees are recoverable from an LM's contextualized
embeddings. Across 46 typologically and architecturally diverse LM-language
pairs, our probing approach predicts the best LM choice 79% of the time using
orders of magnitude less compute than training a full parser. Within this
study, we identify and analyze one recently proposed decoupled LM - RemBERT -
and find it strikingly contains less inherent dependency information, but often
yields the best parser after full fine-tuning. Without this outlier our
approach identifies the best LM in 89% of cases.

通过衡量语言模型上下文嵌入中恢复标记树的程度，我们提出使用考察探测来对语言模型进行排序，从而确定适合特定语言的最佳模型选择。在 46 种不同类型和结构的语言模型 - 语言对中，我们的探测方法预测最佳语言模型选择的准确率为 79％，比训练完整解析器需要更少的计算量。在本研究中，我们发现 RemBERT 是一个最近提出的解耦合语言模型，它显著包含较少的固有依赖信息，但通常在完全微调后可以产生最佳的解析器结果。在排除这个异常值后，我们的方法在 89％的情况下确定最佳的语言模型选择。