Large language models (LLMs) have exhibited considerable cross-lingual
generalization abilities, whereby they implicitly transfer knowledge across
languages. However, the transfer is not equally successful for all languages,
especially for low-resource ones, which poses an ongoing challenge. It is
unclear whether we have reached the limits of implicit cross-lingual
generalization and if explicit knowledge transfer is viable. In this paper, we
investigate the potential for explicitly aligning conceptual correspondence
between languages to enhance cross-lingual generalization. Using the syntactic
aspect of language as a testbed, our analyses of 43 languages reveal a high
degree of alignability among the spaces of structural concepts within each
language for both encoder-only and decoder-only LLMs. We then propose a
meta-learning-based method to learn to align conceptual spaces of different
languages, which facilitates zero-shot and few-shot generalization in concept
classification and also offers insights into the cross-lingual in-context
learning phenomenon. Experiments on syntactic analysis tasks show that our
approach achieves competitive results with state-of-the-art methods and narrows
the performance gap between languages, particularly benefiting those with
limited resources.

利用语言的语法方面作为测试基础，我们的分析揭示了对于仅编码器和仅解码器的大型语言模型，语言的结构概念空间之间具有高度的对齐度。然后，我们提出了一种基于元学习的方法来学习不同语言之间的概念空间对齐，从而实现零样本学习和少样本学习，并增进了对跨语境情景的学习现象的理解。在语法分析任务上的实验表明，我们的方法与最先进的方法相比取得了竞争性的结果，并缩小了语言之间的性能差距，特别有助于那些资源有限的语言。

变压器语言模型中的结构概念是否普遍？迈向可解释的跨语言泛化

Are Structural Concepts Universal in Transformer Language Models?  Towards Interpretable Cross-Lingual Generalization

Large language models (LLMs) can acquire strong code-generation capabilities
through few-shot learning. In contrast, supervised fine-tuning is still needed
for smaller models to achieve good performance. Such fine-tuning demands a
large number of task-specific NL-code pairs, which are expensive to obtain. In
this paper, we attempt to transfer the code generation ability of an LLM to a
smaller model with the aid of weakly-supervised data. More specifically, we
propose explicit knowledge transfer (EKT), which uses the few-shot capabilities
of a teacher LLM to create NL-code pairs that we then filter for correctness
and fine-tune the student on. We evaluate EKT on the task of generating code
solutions to math word problems from the GSM8k dataset. We find that EKT not
only yields better performance than training with expert iteration, but also
outperforms knowledge distillation, another form of knowledge transfer. A
GPT-Neo 1.3B model trained using EKT with a GPT-J teacher achieves a 12.4%
pass@100 on GSM8k, while the same student and teacher trained with knowledge
distillation yield only a 3.7% pass@100. We also show that it is possible for a
student model to outperform the teacher using EKT.

本研究提出了一种显式知识转移（EKT）方法，利用大型语言模型（LLMs）的少量学习能力来创建 NL-code 对，过滤正确性并对学生模型进行微调，以实现在数学问题求解中生成代码解决方案的任务。实验结果表明，使用 EKT 训练的 GPT-Neo 1.3B 模型具有比使用专家迭代训练和知识蒸馏更好的性能，并且学生模型使用 EKT 优于教师模型的可能性。