The lottery ticket hypothesis (LTH) has increased attention to pruning neural
networks at initialization. We study this problem in the linear setting. We
show that finding a sparse mask at initialization is equivalent to the
sketching problem introduced for efficient matrix multiplication. This gives us
tools to analyze the LTH problem and gain insights into it. Specifically, using
the mask found at initialization, we bound the approximation error of the
pruned linear model at the end of training. We theoretically justify previous
empirical evidence that the search for sparse networks may be data independent.
By using the sketching perspective, we suggest a generic improvement to
existing algorithms for pruning at initialization, which we show to be
beneficial in the data-independent case.

研究了在线性设置中用于剪枝神经网络的 lottery ticket hypothesis (LTH)，发现找到一个稀疏的 mask 等价于为高效矩阵乘法引入的 sketching 问题，并以此作为工具来分析 LTH 问题并提出通用改进算法。

初始化剪枝：基于草图视角

Pruning at Initialization -- A Sketching Perspective

Parameter-efficient fine-tuning (PEFT) of pre-trained language models has
recently demonstrated remarkable achievements, effectively matching the
performance of full fine-tuning while utilizing significantly fewer trainable
parameters, and consequently addressing the storage and communication
constraints. Nonetheless, various PEFT methods are limited by their inherent
characteristics. In the case of sparse fine-tuning, which involves modifying
only a small subset of the existing parameters, the selection of fine-tuned
parameters is task- and domain-specific, making it unsuitable for federated
learning. On the other hand, PEFT methods with adding new parameters typically
introduce additional inference latency. In this paper, we demonstrate the
feasibility of generating a sparse mask in a task-agnostic manner, wherein all
downstream tasks share a common mask. Our approach, which relies solely on the
magnitude information of pre-trained parameters, surpasses existing
methodologies by a significant margin when evaluated on the GLUE benchmark.
Additionally, we introduce a novel adapter technique that directly applies the
adapter to pre-trained parameters instead of the hidden representation, thereby
achieving identical inference speed to that of full fine-tuning. Through
extensive experiments, our proposed method attains a new state-of-the-art
outcome in terms of both performance and storage efficiency, storing only 0.03%
parameters of full fine-tuning.

本文提出了一种任务不可知的生成稀疏掩码的方法，仅使用预训练参数的振幅信息，可以显著提高性能和存储效率，并引入了一种新颖的适配器技术，可以直接应用于预训练参数，与全细调速度相同。

无需增加延迟的参数高效微调

Parameter-Efficient Fine-Tuning without Introducing New Latency

Federated learning (FL) is vulnerable to heterogeneously distributed data,
since a common global model in FL may not adapt to the heterogeneous data
distribution of each user. To counter this issue, personalized FL (PFL) was
proposed to produce dedicated local models for each individual user. However,
PFL is far from its maturity, because existing PFL solutions either demonstrate
unsatisfactory generalization towards different model architectures or cost
enormous extra computation and memory. In this work, we propose federated
learning with personalized sparse mask (FedSpa), a novel PFL scheme that
employs personalized sparse masks to customize sparse local models on the edge.
Instead of training an intact (or dense) PFL model, FedSpa only maintains a
fixed number of active parameters throughout training (aka sparse-to-sparse
training), which enables users' models to achieve personalization with cheap
communication, computation, and memory cost. We theoretically show that the
iterates obtained by FedSpa converge to the local minimizer of the formulated
SPFL problem at rate of $\mathcal{O}(\frac{1}{\sqrt{T}})$. Comprehensive
experiments demonstrate that FedSpa significantly saves communication and
computation costs, while simultaneously achieves higher model accuracy and
faster convergence speed against several state-of-the-art PFL methods.

本论文提出一种名为 FedSpa 的新型 PFL 方案，它采用个性化稀疏掩码来在边缘上自定义稀疏的本地模型，理论和实验表明，FedSpa 不仅节省了通信和计算成本，而且在模型准确性和收敛速度方面表现优异。

使用稀疏本地模型实现个性化联邦学习

Achieving Personalized Federated Learning with Sparse Local Models

During typical gradient-based training of deep neural networks, all of the
model's parameters are updated at each iteration. Recent work has shown that it
is possible to update only a small subset of the model's parameters during
training, which can alleviate storage and communication requirements. In this
paper, we show that it is possible to induce a fixed sparse mask on the model's
parameters that selects a subset to update over many iterations. Our method
constructs the mask out of the $k$ parameters with the largest Fisher
information as a simple approximation as to which parameters are most important
for the task at hand. In experiments on parameter-efficient transfer learning
and distributed training, we show that our approach matches or exceeds the
performance of other methods for training with sparse updates while being more
efficient in terms of memory usage and communication costs. We release our code
publicly to promote further applications of our approach.

本文提出了种方法，在深度神经网络的训练过程中，仅更新极少参数，通过一个由具有最大 Fisher 信息的 k 个参数组成的稀疏掩码选择一个参数子集进行多次迭代更新。实验证明，该方法在参数有效转移学习和分布式训练方面的表现可与其他方法相匹配或超过，并且使用的存储和通信资源更少。