Parameter regularization or allocation methods are effective in overcoming
catastrophic forgetting in lifelong learning. However, they solve all tasks in
a sequence uniformly and ignore the differences in the learning difficulty of
different tasks. So parameter regularization methods face significant
forgetting when learning a new task very different from learned tasks, and
parameter allocation methods face unnecessary parameter overhead when learning
simple tasks. In this paper, we propose the Parameter Allocation &
Regularization (PAR), which adaptively select an appropriate strategy for each
task from parameter allocation and regularization based on its learning
difficulty. A task is easy for a model that has learned tasks related to it and
vice versa. We propose a divergence estimation method based on the
Nearest-Prototype distance to measure the task relatedness using only features
of the new task. Moreover, we propose a time-efficient relatedness-aware
sampling-based architecture search strategy to reduce the parameter overhead
for allocation. Experimental results on multiple benchmarks demonstrate that,
compared with SOTAs, our method is scalable and significantly reduces the
model's redundancy while improving the model's performance. Further qualitative
analysis indicates that PAR obtains reasonable task-relatedness.

本文提出了参数分配与正则化 (PAR) 方法，以适应任务的学习难度，从参数分配和正则化中自适应地选择适当的策略，并使用基于最近原型距离的发散估计方法来测量任务相关性，通过时间高效的相关性感知采样搜索策略来减少分配的参数开销，实验结果表明，与 SOTAs 相比，我们的方法可扩展且显著减少了模型的冗余，同时提高了模型的性能，并进一步说明 PAR 获得了合理的任务相关性。

面向任务难度感知的生命周期学习参数分配和正则化

Task Difficulty Aware Parameter Allocation & Regularization for Lifelong  Learning

We re-evaluate the standard practice of sharing weights between input and
output embeddings in state-of-the-art pre-trained language models. We show that
decoupled embeddings provide increased modeling flexibility, allowing us to
significantly improve the efficiency of parameter allocation in the input
embedding of multilingual models. By reallocating the input embedding
parameters in the Transformer layers, we achieve dramatically better
performance on standard natural language understanding tasks with the same
number of parameters during fine-tuning. We also show that allocating
additional capacity to the output embedding provides benefits to the model that
persist through the fine-tuning stage even though the output embedding is
discarded after pre-training. Our analysis shows that larger output embeddings
prevent the model's last layers from overspecializing to the pre-training task
and encourage Transformer representations to be more general and more
transferable to other tasks and languages. Harnessing these findings, we are
able to train models that achieve strong performance on the XTREME benchmark
without increasing the number of parameters at the fine-tuning stage.

重新评估了先进的预训练语言模型中共享输入和输出嵌入权重的标准做法，表明解耦的嵌入提供了更好的建模灵活性，允许我们在多语言模型的输入嵌入的参数分配方面显着提高参数分配效率。通过在 Transformer 层中重新分配输入嵌入参数，我们在保持微调期间相同的参数数量的情况下，实现了标准自然语言理解任务的极大性能改进。Allocate 额外的容量来提供模型的输出嵌入即使在预训练后被丢弃，也能在微调阶段带来好处。我们的分析表明，更大的输出嵌入可以防止模型的最后几层过度专注于预训练任务，并鼓励 Transformer 表示更加普遍和更易于传递到其他任务和语言。利用这些发现，我们能够训练出在 XTREME 基准上表现出色的模型，而不增加微调阶段的参数数量。

重新思考预训练语言模型中的嵌入耦合

Rethinking embedding coupling in pre-trained language models

To leverage data and computation capabilities of mobile devices, machine
learning algorithms are deployed at the network edge for training artificial
intelligence (AI) models, resulting in the new paradigm of edge learning. In
this paper, we consider the framework of partitioned edge learning for
iteratively training a large-scale model using many resource-constrained
devices (called workers). To this end, in each iteration, the model is
dynamically partitioned into parametric blocks, which are downloaded to worker
groups for updating using data subsets. Then, the local updates are uploaded to
and cascaded by the server for updating a global model. To reduce resource
usage by minimizing the total learning-and-communication latency, this work
focuses on the novel joint design of parameter (computation load) allocation
and bandwidth allocation (for downloading and uploading). Two design approaches
are adopted. First, a practical sequential approach, called partially
integrated parameter-and-bandwidth allocation (PABA), yields two schemes,
namely bandwidth aware parameter allocation and parameter aware bandwidth
allocation. The former minimizes the load for the slowest (in computing) of
worker groups, each training a same parametric block. The latter allocates the
largest bandwidth to the worker being the latency bottleneck. Second, PABA are
jointly optimized. Despite its being a nonconvex problem, an efficient and
optimal solution algorithm is derived by intelligently nesting a bisection
search and solving a convex problem. Experimental results using real data
demonstrate that integrating PABA can substantially improve the performance of
partitioned edge learning in terms of latency (by e.g., 46%) and accuracy (by
e.g., 4%).

本文提出了一个基于有限资源设备的边缘学习框架，其中参数以及带宽分配共同优化，实验结果表明此方法能够显著提高分区式边缘学习的性能。