Federated learning (FL) is a heavily promoted approach for training ML models
on sensitive data, e.g., text typed by users on their smartphones. FL is
expressly designed for training on data that are unbalanced and non-iid across
the participants. To ensure privacy and integrity of the fedeated model, latest
FL approaches use differential privacy or robust aggregation.
We look at FL from the \emph{local} viewpoint of an individual participant
and ask: (1) do participants have an incentive to participate in FL? (2) how
can participants \emph{individually} improve the quality of their local models,
without re-designing the FL framework and/or involving other participants?
First, we show that on standard tasks such as next-word prediction, many
participants gain no benefit from FL because the federated model is less
accurate on their data than the models they can train locally on their own.
Second, we show that differential privacy and robust aggregation make this
problem worse by further destroying the accuracy of the federated model for
many participants.
Then, we evaluate three techniques for local adaptation of federated models:
fine-tuning, multi-task learning, and knowledge distillation. We analyze where
each is applicable and demonstrate that all participants benefit from local
adaptation. Participants whose local models are poor obtain big accuracy
improvements over conventional FL. Participants whose local models are better
than the federated model\textemdash and who have no incentive to participate in
FL today\textemdash improve less, but sufficiently to make the adapted
federated model better than their local models.

针对联邦学习中参与者局部利益与数据准确度不匹配的问题，本文提出并比较了三种本地适应技术，并指出差分隐私和鲁棒聚合恶化了联邦模型的准确性。这三种技术分别为：微调、多任务学习和知识蒸馏。我们的实验结果表明，所有参与者都从本地适应中受益，并且本地模型表现不佳的参与者通过传统联邦方式得到了大幅提升。

通过本地适配挽救联合学习

Salvaging Federated Learning by Local Adaptation

We introduce a lifelong language learning setup where a model needs to learn
from a stream of text examples without any dataset identifier. We propose an
episodic memory model that performs sparse experience replay and local
adaptation to mitigate catastrophic forgetting in this setup. Experiments on
text classification and question answering demonstrate the complementary
benefits of sparse experience replay and local adaptation to allow the model to
continuously learn from new datasets. We also show that the space complexity of
the episodic memory module can be reduced significantly (~50-90%) by randomly
choosing which examples to store in memory with a minimal decrease in
performance. We consider an episodic memory component as a crucial building
block of general linguistic intelligence and see our model as a first step in
that direction.

提出一种生命化语言学习的模型，其中采用了 “稀疏经验回放” 和 “本地适应” 以减轻灾难性遗忘，并将这种情况应用于文本分类和问题回答中，证明了这种模型的优越性，同时表明通过随机选择存储在内存中的样本可以大大减少经验记忆模块的空间复杂度，并认为经验记忆组件是构建通用语言智能的重要组成部分。

终身语言学习中的情景记忆

Episodic Memory in Lifelong Language Learning

Knowledge graph embedding aims to represent entities and relations in a
large-scale knowledge graph as elements in a continuous vector space. Existing
methods, e.g., TransE and TransH, learn embedding representation by defining a
global margin-based loss function over the data. However, the optimal loss
function is determined during experiments whose parameters are examined among a
closed set of candidates. Moreover, embeddings over two knowledge graphs with
different entities and relations share the same set of candidate loss
functions, ignoring the locality of both graphs. This leads to the limited
performance of embedding related applications. In this paper, we propose a
locally adaptive translation method for knowledge graph embedding, called
TransA, to find the optimal loss function by adaptively determining its margin
over different knowledge graphs. Experiments on two benchmark data sets
demonstrate the superiority of the proposed method, as compared to
the-state-of-the-art ones.

该论文提出了一种名为 TransA 的知识图谱嵌入方法，通过局部自适应方法确定不同知识图谱上的最优损失函数来提高嵌入相关应用的性能，针对两个基准数据集的实验结果表明该方法具有超过现有方法的优越性。