This paper explores the integration of graph knowledge from linguistic
ontologies into multilingual Large Language Models (LLMs) using adapters to
improve performance for low-resource languages (LRLs) in sentiment analysis
(SA) and named entity recognition (NER). Building upon successful
parameter-efficient fine-tuning techniques, such as K-ADAPTER and MAD-X, we
propose a similar approach for incorporating knowledge from multilingual
graphs, connecting concepts in various languages with each other through
linguistic relationships, into multilingual LLMs for LRLs. Specifically, we
focus on eight LRLs -- Maltese, Bulgarian, Indonesian, Nepali, Javanese,
Uyghur, Tibetan, and Sinhala -- and employ language-specific adapters
fine-tuned on data extracted from the language-specific section of ConceptNet,
aiming to enable knowledge transfer across the languages covered by the
knowledge graph. We compare various fine-tuning objectives, including standard
Masked Language Modeling (MLM), MLM with full-word masking, and MLM with
targeted masking, to analyse their effectiveness in learning and integrating
the extracted graph data. Through empirical evaluation on language-specific
tasks, we assess how structured graph knowledge affects the performance of
multilingual LLMs for LRLs in SA and NER, providing insights into the potential
benefits of adapting language models for low-resource scenarios.

该研究探讨了将语义本体中的图谱知识整合到多语言大型语言模型中，通过适配器提高低资源语言在情感分析和命名实体识别中的性能，对八种低资源语言进行了实证评估，提供了适应低资源情景的语言模型优势的见解。

通过适配器在知识图谱上将多语言 LLMs 调整为资源不足的语言

Adapting Multilingual LLMs to Low-Resource Languages with Knowledge  Graphs via Adapters

In visual tasks, large teacher models capture essential features and deep
information, enhancing performance. However, distilling this information into
smaller student models often leads to performance loss due to structural
differences and capacity limitations. To tackle this, we propose a distillation
framework based on graph knowledge, including a multi-level feature alignment
strategy and an attention-guided mechanism to provide a targeted learning
trajectory for the student model. We emphasize spectral embedding (SE) as a key
technique in our distillation process, which merges the student's feature space
with the relational knowledge and structural complexities similar to the
teacher network. This method captures the teacher's understanding in a
graph-based representation, enabling the student model to more accurately mimic
the complex structural dependencies present in the teacher model. Compared to
methods that focus only on specific distillation areas, our strategy not only
considers key features within the teacher model but also endeavors to capture
the relationships and interactions among feature sets, encoding these complex
pieces of information into a graph structure to understand and utilize the
dynamic relationships among these pieces of information from a global
perspective. Experiments show that our method outperforms previous feature
distillation methods on the CIFAR-100, MS-COCO, and Pascal VOC datasets,
proving its efficiency and applicability.

基于图知识的蒸馏方法结合多级特征对齐策略和注意力引导机制，利用谱嵌入的技术将学生模型的特征空间与教师网络的关系和结构复杂性相融合，通过全局视角理解和利用特征集之间的动态关系，从而更准确地模仿教师模型的复杂结构性依赖关系，实验证明该方法在 CIFAR-100、MS-COCO 和 Pascal VOC 数据集上的效果优于以往的特征蒸馏方法，证明其有效性和适用性。