Federated Learning (FL) is a privacy-preserving distributed learning approach
that is rapidly developing in an era where privacy protection is increasingly
valued. It is this rapid development trend, along with the continuous emergence
of new demands for FL in the real world, that prompts us to focus on a very
important problem: Federated Learning with New Knowledge. The primary challenge
here is to effectively incorporate various new knowledge into existing FL
systems and evolve these systems to reduce costs, extend their lifespan, and
facilitate sustainable development. In this paper, we systematically define the
main sources of new knowledge in FL, including new features, tasks, models, and
algorithms. For each source, we thoroughly analyze and discuss how to
incorporate new knowledge into existing FL systems and examine the impact of
the form and timing of new knowledge arrival on the incorporation process.
Furthermore, we comprehensively discuss the potential future directions for FL
with new knowledge, considering a variety of factors such as scenario setups,
efficiency, and security. There is also a continuously updating repository for
this topic: this https URL

本文系统性地定义了 FL 中新知识的主要来源，包括新特征、任务、模型和算法，详细分析讨论了如何将新知识纳入现有的 FL 系统，并研究了新知识到达的形式和时机对纳入过程的影响，讨论了 FL 中新知识的潜在未来方向。

具备新知识的联邦学习：基础、进展与未来

Federated Learning with New Knowledge: Fundamentals, Advances, and  Futures

With the rapid development of NLP, large-scale language models (LLMs) excel
in various tasks across multiple domains now. However, existing benchmarks may
not adequately measure these models' capabilities, especially when faced with
new knowledge. In this paper, we address the lack of benchmarks to evaluate
LLMs' ability to handle new knowledge, an important and challenging aspect in
the rapidly evolving world. We propose an approach called KnowGen that
generates new knowledge by altering existing entity attributes and
relationships, resulting in artificial entities that are distinct from
real-world entities. With KnowGen, we introduce a benchmark named ALCUNA to
assess LLMs' abilities in knowledge understanding, differentiation, and
association. We benchmark several LLMs, reveals that their performance in face
of new knowledge is not satisfactory, particularly in reasoning between new and
internal knowledge. We also explore the impact of entity similarity on the
model's understanding of entity knowledge and the influence of contextual
entities. We appeal to the need for caution when using LLMs in new scenarios or
with new knowledge, and hope that our benchmarks can help drive the development
of LLMs in face of new knowledge.

提出了一个名为 KnowGen 的方法来生成新知识，并引入了一个名为 ALCUNA 的基准来评估大规模语言模型在处理新知识方面的能力，发现它们在处理新知识的推理方面表现不佳，并探索了实体相似性对模型对实体知识的理解以及上下文实体的影响。

ALCUNA：大型语言模型迎接新知识

ALCUNA: Large Language Models Meet New Knowledge

Language models typically need to be trained or finetuned in order to acquire
new knowledge, which involves updating their weights. We instead envision
language models that can simply read and memorize new data at inference time,
thus acquiring new knowledge immediately. In this work, we extend language
models with the ability to memorize the internal representations of past
inputs. We demonstrate that an approximate kNN lookup into a non-differentiable
memory of recent (key, value) pairs improves language modeling across various
benchmarks and tasks, including generic webtext (C4), math papers (arXiv),
books (PG-19), code (Github), as well as formal theorems (Isabelle). We show
that the performance steadily improves when we increase the size of memory up
to 262K tokens. On benchmarks including code and mathematics, we find that the
model is capable of making use of newly defined functions and theorems during
test time.

本文介绍了一种使用内部储存器实现直接读取并记忆新数据的语言模型，在多个基准测试和任务中展示了近似 kNN 查找技术，着重测试了代码和数学等领域，并证明了随着储存器大小的增加，性能将稳步提高。