Multilingual large-scale Pretrained Language Models (PLMs) have been shown to
store considerable amounts of factual knowledge, but large variations are
observed across languages. With the ultimate goal of ensuring that users with
different language backgrounds obtain consistent feedback from the same model,
we study the cross-lingual consistency (CLC) of factual knowledge in various
multilingual PLMs. To this end, we propose a Ranking-based Consistency (RankC)
metric to evaluate knowledge consistency across languages independently from
accuracy. Using this metric, we conduct an in-depth analysis of the determining
factors for CLC, both at model level and at language-pair level. Among other
results, we find that increasing model size leads to higher factual probing
accuracy in most languages, but does not improve cross-lingual consistency.
Finally, we conduct a case study on CLC when new factual associations are
inserted in the PLMs via model editing. Results on a small sample of facts
inserted in English reveal a clear pattern whereby the new piece of knowledge
transfers only to languages with which English has a high RankC score.

对多语言大规模预训练语言模型进行研究，发现不同语言之间的事实知识存在显著差异。为了确保具有不同语言背景的用户从同一模型中获得一致的反馈，我们提出了一种基于排名的一致性评估指标，并对模型层面和语言对层面的一致性决定因素进行了深入分析。研究结果表明，增加模型规模可以提高大多数语言的事实检测准确性，但无法改善跨语言一致性。通过在模型编辑中插入新的事实关联进行的案例研究显示，新的知识仅传递给与英语具有较高排名一致性得分的语言。