Recent research in federated large language models (LLMs) has primarily
focused on enabling clients to fine-tune their locally deployed homogeneous
LLMs collaboratively or on transferring knowledge from server-based LLMs to
small language models (SLMs) at downstream clients. However, a significant gap
remains in the simultaneous mutual enhancement of both the server's LLM and
clients' SLMs. To bridge this gap, we propose FedMKT, a parameter-efficient
federated mutual knowledge transfer framework for large and small language
models. This framework is designed to adaptively transfer knowledge from the
server's LLM to clients' SLMs while concurrently enriching the LLM with
clients' unique domain insights. We facilitate token alignment using minimum
edit distance (MinED) and then selective mutual knowledge transfer between
client-side SLMs and a server-side LLM, aiming to collectively enhance their
performance. Through extensive experiments across three distinct scenarios,
heterogeneous, homogeneous, and one-to-one, we evaluate the effectiveness of
FedMKT using various public LLMs and SLMs on a range of NLP text generation
tasks. Empirical results demonstrate significant performance improvements in
clients' SLMs with the aid of the LLM. Furthermore, the LLM optimized by FedMKT
achieves a performance comparable to that achieved through direct fine-tuning
based on clients' data, highlighting the effectiveness and adaptability of
FedMKT.

FedMKT 是一个参数高效的联邦互相知识转移框架，旨在同时改进服务器端和客户端的大型和小型语言模型，并通过最小编辑距离实现令牌对齐和选择性的互相知识转移，在各种 NLP 文本生成任务中显著提高客户端的小型语言模型性能，并通过 FedMKT 优化的大型语言模型实现了与基于客户端数据直接微调相当的性能，凸显了 FedMKT 的效果和适应性。