To address the communication bottleneck challenge in distributed learning,
our work introduces a novel two-stage quantization strategy designed to enhance
the communication efficiency of distributed Stochastic Gradient Descent (SGD).
The proposed method initially employs truncation to mitigate the impact of
long-tail noise, followed by a non-uniform quantization of the post-truncation
gradients based on their statistical characteristics. We provide a
comprehensive convergence analysis of the quantized distributed SGD,
establishing theoretical guarantees for its performance. Furthermore, by
minimizing the convergence error, we derive optimal closed-form solutions for
the truncation threshold and non-uniform quantization levels under given
communication constraints. Both theoretical insights and extensive experimental
evaluations demonstrate that our proposed algorithm outperforms existing
quantization schemes, striking a superior balance between communication
efficiency and convergence performance.

为了解决分布式学习中的通信瓶颈挑战，本研究引入了一种新的两阶段量化策略，旨在增强分布式随机梯度下降（SGD）的通信效率。通过截断来减轻长尾噪声的影响，随后根据梯度的统计特征进行非均匀量化。我们为量化的分布式 SGD 提供了全面的收敛性分析，为其性能提供了理论保证。此外，通过最小化收敛误差，我们推导出了在给定通信约束下的截断阈值和非均匀量化水平的最优闭式解。理论洞察力和广泛的实验评估表明，我们的算法优于现有的量化方案，在通信效率和收敛性能之间达到了更优的平衡。

分布式 SGD 的截断非均匀量化

Truncated Non-Uniform Quantization for Distributed SGD

Hashing is at the heart of large-scale image similarity search, and recent
methods have been substantially improved through deep learning techniques. Such
algorithms typically learn continuous embeddings of the data. To avoid a
subsequent costly binarization step, a common solution is to employ loss
functions that combine a similarity learning term (to ensure similar images are
grouped to nearby embeddings) and a quantization penalty term (to ensure that
the embedding entries are close to binarized entries, e.g., -1 or 1). Still,
the interaction between these two terms can make learning harder and the
embeddings worse. We propose an alternative quantization strategy that
decomposes the learning problem in two stages: first, perform similarity
learning over the embedding space with no quantization; second, find an optimal
orthogonal transformation of the embeddings so each coordinate of the embedding
is close to its sign, and then quantize the transformed embedding through the
sign function. In the second step, we parametrize orthogonal transformations
using Householder matrices to efficiently leverage stochastic gradient descent.
Since similarity measures are usually invariant under orthogonal
transformations, this quantization strategy comes at no cost in terms of
performance. The resulting algorithm is unsupervised, fast, hyperparameter-free
and can be run on top of any existing deep hashing or metric learning
algorithm. We provide extensive experimental results showing that this approach
leads to state-of-the-art performance on widely used image datasets, and,
unlike other quantization strategies, brings consistent improvements in
performance to existing deep hashing algorithms.

通过提出一种替代的量化策略，该论文通过在嵌入空间上进行相似性学习，并采用正交变换优化嵌入的每个坐标，然后通过符号函数将转换后的嵌入量化为二进制，从而提高了大规模图像相似性搜索的性能。

基于 Householder 量化的深度哈希

Deep Hashing via Householder Quantization

We show that the majority of the inference computations for large generative
models such as LLaMA and OPT can be performed with both weights and activations
being cast to 4 bits, in a way that leads to practical speedups while at the
same time maintaining good accuracy. We achieve this via a hybrid quantization
strategy called QUIK, which compresses most of the weights and activations to
4-bit, while keeping some outlier weights and activations in higher-precision.
Crucially, our scheme is designed with computational efficiency in mind: we
provide GPU kernels with highly-efficient layer-wise runtimes, which lead to
practical end-to-end throughput improvements of up to 3.1x relative to FP16
execution. Code and models are provided at this https URL

大多数大型生成模型的推理计算可以通过将权重和激活值均转换为 4 位来加速计算，同时保持良好的准确性；我们通过名为 QUIK 的混合量化策略实现这一目标，该策略将大多数权重和激活值压缩为 4 位，将一些异常值保留在较高精度；关键是，我们的方案专注于计算效率，提供高效的逐层 GPU 内核，相对于 FP16 执行，端到端的吞吐量可提高最多 3.1 倍。