Recent research in federated large language models (LLMs) has primarily
focused on enabling clients to fine-tune their locally deployed homogeneous
LLMs collaboratively or on transferring knowledge from server-based LLMs to
small language models (SLMs) at downstream clients. However, a significant gap
remains in the simultaneous mutual enhancement of both the server's LLM and
clients' SLMs. To bridge this gap, we propose FedMKT, a parameter-efficient
federated mutual knowledge transfer framework for large and small language
models. This framework is designed to adaptively transfer knowledge from the
server's LLM to clients' SLMs while concurrently enriching the LLM with
clients' unique domain insights. We facilitate token alignment using minimum
edit distance (MinED) and then selective mutual knowledge transfer between
client-side SLMs and a server-side LLM, aiming to collectively enhance their
performance. Through extensive experiments across three distinct scenarios,
heterogeneous, homogeneous, and one-to-one, we evaluate the effectiveness of
FedMKT using various public LLMs and SLMs on a range of NLP text generation
tasks. Empirical results demonstrate significant performance improvements in
clients' SLMs with the aid of the LLM. Furthermore, the LLM optimized by FedMKT
achieves a performance comparable to that achieved through direct fine-tuning
based on clients' data, highlighting the effectiveness and adaptability of
FedMKT.

FedMKT 是一个参数高效的联邦互相知识转移框架，旨在同时改进服务器端和客户端的大型和小型语言模型，并通过最小编辑距离实现令牌对齐和选择性的互相知识转移，在各种 NLP 文本生成任务中显著提高客户端的小型语言模型性能，并通过 FedMKT 优化的大型语言模型实现了与基于客户端数据直接微调相当的性能，凸显了 FedMKT 的效果和适应性。

FedMKT：大型和小型语言模型的联邦互助知识传输

FedMKT: Federated Mutual Knowledge Transfer for Large and Small Language  Models

Generative models, widely utilized in various applications, can often
struggle with prompts corresponding to partial tokens. This struggle stems from
tokenization, where partial tokens fall out of distribution during inference,
leading to incorrect or nonsensical outputs. This paper examines a technique to
alleviate the tokenization artifact on text completion in generative models,
maintaining performance even in regular non-subword cases. The method, termed
token alignment, involves backtracking to the last complete tokens and ensuring
the model's generation aligns with the prompt. This approach showcases marked
improvement across many partial token scenarios, including nuanced cases like
space-prefix and partial indentation, with only a minor time increase. The
technique and analysis detailed in this paper contribute to the continuous
advancement of generative models in handling partial inputs, bearing relevance
for applications like code completion and text autocompletion.

该论文研究了一种通过回溯和对齐机制来解决生成模型中的标记化问题，从而提高模型在处理部分输入时的性能。该方法对包括代码补全和文本自动完成在内的多种场景都有显著改善，同时只有轻微的时间成本增加。