This paper proposes a novel communication-efficient split learning (SL)
framework, named SplitFC, which reduces the communication overhead required for
transmitting intermediate feature and gradient vectors during the SL training
process. The key idea of SplitFC is to leverage different dispersion degrees
exhibited in the columns of the matrices. SplitFC incorporates two compression
strategies: (i) adaptive feature-wise dropout and (ii) adaptive feature-wise
quantization. In the first strategy, the intermediate feature vectors are
dropped with adaptive dropout probabilities determined based on the standard
deviation of these vectors. Then, by the chain rule, the intermediate gradient
vectors associated with the dropped feature vectors are also dropped. In the
second strategy, the non-dropped intermediate feature and gradient vectors are
quantized using adaptive quantization levels determined based on the ranges of
the vectors. To minimize the quantization error, the optimal quantization
levels of this strategy are derived in a closed-form expression. Simulation
results on the MNIST, CIFAR-10, and CelebA datasets demonstrate that SplitFC
provides more than a 5.6% increase in classification accuracy compared to
state-of-the-art SL frameworks, while they require 320 times less communication
overhead compared to the vanilla SL framework without compression.

本论文提出了一种新颖的通信高效的分离学习（Split Learning）框架，名为 SplitFC，它通过减少 SL 训练过程中传输中间特征向量和梯度向量所需的通信开销。SplitFC 使用两种压缩策略：自适应特征丢弃和自适应特征量化，其实现方式分别是根据向量标准差确定自适应丢弃概率和根据向量范围确定自适应量化级别。实验结果表明，与最先进的 SL 框架相比，SplitFC 在 MNIST、CIFAR-10 和 CelebA 数据集上提供了超过 5.6％的分类准确性提升，同时所需通信开销比不使用压缩的基础 SL 框架少 320 倍。

自适应分特征压缩的通信高效分担学习

Communication-Efficient Split Learning via Adaptive Feature-Wise  Compression

Deep neural networks have proved hugely successful, achieving human-like
performance on a variety of tasks. However, they are also computationally
expensive, which has motivated the development of model compression techniques
which reduce the resource consumption associated with deep learning models.
Nevertheless, recent studies have suggested that model compression can have an
adverse effect on algorithmic fairness, amplifying existing biases in machine
learning models. With this project we aim to extend those studies to the
context of facial expression recognition. To do that, we set up a neural
network classifier to perform facial expression recognition and implement
several model compression techniques on top of it. We then run experiments on
two facial expression datasets, namely the Extended Cohn-Kanade Dataset (CK+DB)
and the Real-World Affective Faces Database (RAF-DB), to examine the individual
and combined effect that compression techniques have on the model size,
accuracy and fairness. Our experimental results show that: (i) Compression and
quantisation achieve significant reduction in model size with minimal impact on
overall accuracy for both CK+DB and RAF-DB; (ii) in terms of model accuracy,
the classifier trained and tested on RAF-DB seems more robust to compression
compared to the CK+ DB; (iii) for RAF-DB, the different compression strategies
do not seem to increase the gap in predictive performance across the sensitive
attributes of gender, race and age which is in contrast with the results on the
CK+DB, where compression seems to amplify existing biases for gender. We
analyse the results and discuss the potential reasons for our findings.

该研究研究了模型压缩技术对面部表情识别模型大小、准确性和公平性的影响，结果表明针对性的技术对模型大小有明显减小效果，对准确性几乎没有影响；对于不同的数据集，受敏感属性的影响不同，作者们对结果进行了分析并讨论了发现的潜在原因。

模型压缩对面部表情识别公平性的影响

The Effect of Model Compression on Fairness in Facial Expression  Recognition

First proposed by Seide (2014) as a heuristic, error feedback (EF) is a very
popular mechanism for enforcing convergence of distributed gradient-based
optimization methods enhanced with communication compression strategies based
on the application of contractive compression operators. However, existing
theory of EF relies on very strong assumptions (e.g., bounded gradients), and
provides pessimistic convergence rates (e.g., while the best known rate for EF
in the smooth nonconvex regime, and when full gradients are compressed, is
$O(1/T^{2/3})$, the rate of gradient descent in the same regime is $O(1/T)$).
Recently, Richt\'{a}rik et al. (2021) proposed a new error feedback mechanism,
EF21, based on the construction of a Markov compressor induced by a contractive
compressor. EF21 removes the aforementioned theoretical deficiencies of EF and
at the same time works better in practice. In this work we propose six
practical extensions of EF21, all supported by strong convergence theory:
partial participation, stochastic approximation, variance reduction, proximal
setting, momentum and bidirectional compression. Several of these techniques
were never analyzed in conjunction with EF before, and in cases where they were
(e.g., bidirectional compression), our rates are vastly superior.

本文提出了 6 种有效的实用扩展 EF21，并分别分析它们的强收敛理论，这些扩展可以与 EF 一起使用，采用压缩策略和带有 Markov 压缩器的新的错误反馈机制可以更好地优化分布式梯度下降算法。

现代误差反馈的实用算法扩展 - EF21 升级版

EF21 with Bells & Whistles: Practical Algorithmic Extensions of Modern  Error Feedback

In recent years, great progress has been made in a variety of application
domains thanks to the development of increasingly deeper neural networks.
Unfortunately, the huge number of units of these networks makes them expensive
both computationally and memory-wise. To overcome this, exploiting the fact
that deep networks are over-parametrized, several compression strategies have
been proposed. These methods, however, typically start from a network that has
been trained in a standard manner, without considering such a future
compression. In this paper, we propose to explicitly account for compression in
the training process. To this end, we introduce a regularizer that encourages
the parameter matrix of each layer to have low rank during training. We show
that accounting for compression during training allows us to learn much more
compact, yet at least as effective, models than state-of-the-art compression
techniques.

本文提出在训练过程中明确考虑模型压缩，通过引入低秩正则化项使每层参数矩阵的秩尽量小，实现更有效的模型压缩。实验证明，这一方法比现有的压缩技术更加高效。