With the proliferation of large pre-trained language models (PLMs),
fine-tuning all model parameters becomes increasingly inefficient, particularly
when dealing with numerous downstream tasks that entail substantial training
and storage costs. Several approaches aimed at achieving parameter-efficient
fine-tuning (PEFT) have been proposed. Among them, Low-Rank Adaptation (LoRA)
stands out as an archetypal method, incorporating trainable rank decomposition
matrices into each target module. Nevertheless, LoRA does not consider the
varying importance of each layer. To address these challenges, we introduce
PRILoRA, which linearly allocates a different rank for each layer, in an
increasing manner, and performs pruning throughout the training process,
considering both the temporary magnitude of weights and the accumulated
statistics of the input to any given layer. We validate the effectiveness of
PRILoRA through extensive experiments on eight GLUE benchmarks, setting a new
state of the art.

PRILoRA 通过在线性分配不同的秩给每一层并在训练过程中进行剪枝，考虑到权重的临时大小和给定层的输入的累积统计信息，验证了其在八个 GLUE 基准测试中的有效性，取得了最新的技术成果。

PRILoRA：修剪和增大秩的低秩适应

PRILoRA: Pruned and Rank-Increasing Low-Rank Adaptation

Sparsely-activated Mixture-of-experts (MoE) models allow the number of
parameters to greatly increase while keeping the amount of computation for a
given token or a given sample unchanged. However, a poor expert routing
strategy (e.g. one resulting in load imbalance) can cause certain experts to be
under-trained, leading to an expert being under or over-specialized. Prior work
allocates a fixed number of experts to each token using a top-k function
regardless of the relative importance of different tokens. To address this, we
propose a heterogeneous mixture-of-experts employing an expert choice method.
Instead of letting tokens select the top-k experts, we have experts selecting
the top-k tokens. As a result, each token can be routed to a variable number of
experts and each expert can have a fixed bucket size. We systematically study
pre-training speedups using the same computational resources of the Switch
Transformer top-1 and GShard top-2 gating of prior work and find that our
method improves training convergence time by more than 2x. For the same
computational cost, our method demonstrates higher performance in fine-tuning
11 selected tasks in the GLUE and SuperGLUE benchmarks. For a smaller
activation cost, our method outperforms the T5 dense model in 7 out of the 11
tasks.

我们提出了一种基于专家选择的异构专家混合模型，通过让专家选择前 k 个标记来分配变量数量的专家，从而提高了训练收敛速度，并在 GLUE 基准测试中取得更高的性能。

带有专家路由选择的专家混合模型

Mixture-of-Experts with Expert Choice Routing

Transformer models are permutation equivariant. To supply the order and type
information of the input tokens, position and segment embeddings are usually
added to the input. Recent works proposed variations of positional encodings
with relative position encodings achieving better performance. Our analysis
shows that the gain actually comes from moving positional information to
attention layer from the input. Motivated by this, we introduce Decoupled
Positional Attention for Transformers (DIET), a simple yet effective mechanism
to encode position and segment information into the Transformer models. The
proposed method has faster training and inference time, while achieving
competitive performance on GLUE, XTREME and WMT benchmarks. We further
generalize our method to long-range transformers and show performance gain.

本文介绍了一种新的机制 ——Decoupled Positional Attention，将位置和段信息编码为 Transformer 模型，提高了训练和推理效率，在 GLUE、XTREME 和 WMT 基准测试中实现了竞争性表现，并进一步将该方法推广到远程的 transformers，显示了性能提升。