Stateful optimizers maintain gradient statistics over time, e.g., the
exponentially smoothed sum (SGD with momentum) or squared sum (Adam) of past
gradient values. This state can be used to accelerate optimization compared to
plain stochastic gradient descent but uses memory that might otherwise be
allocated to model parameters, thereby limiting the maximum size of models
trained in practice. In this paper, we develop the first optimizers that use
8-bit statistics while maintaining the performance levels of using 32-bit
optimizer states. To overcome the resulting computational, quantization, and
stability challenges, we develop block-wise dynamic quantization. Block-wise
quantization divides input tensors into smaller blocks that are independently
quantized. Each block is processed in parallel across cores, yielding faster
optimization and high precision quantization. To maintain stability and
performance, we combine block-wise quantization with two additional changes:
(1) dynamic quantization, a form of non-linear optimization that is precise for
both large and small magnitude values, and (2) a stable embedding layer to
reduce gradient variance that comes from the highly non-uniform distribution of
input tokens in language models. As a result, our 8-bit optimizers maintain
32-bit performance with a small fraction of the memory footprint on a range of
tasks, including 1.5B parameter language modeling, GLUE finetuning, ImageNet
classification, WMT'14 machine translation, MoCo v2 contrastive ImageNet
pretraining+finetuning, and RoBERTa pretraining, without changes to the
original optimizer hyperparameters. We open-source our 8-bit optimizers as a
drop-in replacement that only requires a two-line code change.

本文开发了首个使用 8 位统计量进行训练的优化器，采用了分块动态量化进行加速，同时结合了非线性优化和嵌入层技术以提高精度和稳定性，并在一系列任务中展现了较高的性能和较小的内存占用。

基于块状量化的 8 位优化器

8-bit Optimizers via Block-wise Quantization

In this work, we consider one challenging training time attack by modifying
training data with bounded perturbation, hoping to manipulate the behavior
(both targeted or non-targeted) of any corresponding trained classifier during
test time when facing clean samples. To achieve this, we proposed to use an
auto-encoder-like network to generate the pertubation on the training data
paired with one differentiable system acting as the imaginary victim
classifier. The perturbation generator will learn to update its weights by
watching the training procedure of the imaginary classifier in order to produce
the most harmful and imperceivable noise which in turn will lead the lowest
generalization power for the victim classifier. This can be formulated into a
non-linear equality constrained optimization problem. Unlike GANs, solving such
problem is computationally challenging, we then proposed a simple yet effective
procedure to decouple the alternating updates for the two networks for
stability. The method proposed in this paper can be easily extended to the
label specific setting where the attacker can manipulate the predictions of the
victim classifiers according to some predefined rules rather than only making
wrong predictions. Experiments on various datasets including CIFAR-10 and a
reduced version of ImageNet confirmed the effectiveness of the proposed method
and empirical results showed that, such bounded perturbation have good
transferability regardless of which classifier the victim is actually using on
image data.

本文提出了一种使用类自编码器的网络生成对训练数据加入扰动的新方法，可以对受害分类器的行为（无论是有目的还是非有目的的）进行操纵并降低其泛化能力。通过实验验证，这种方法可以被应用到各种数据集中，并且具有很好的可迁移性。

学习混淆：使用自编码生成训练时对抗数据

Learning to Confuse: Generating Training Time Adversarial Data with  Auto-Encoder

The non-linear optimization method developed by Konnov and Krotov [Automation
and Remote Control 60, 1427 (1999)] has been used previously to extend the
capabilities of optimal control theory from the linear to the non-linear
Schr\"odinger equation [Sklarz and Tannor, Phys. Rev. A 66, 053619 (2002)].
Here we show that based on the Konnov-Krotov method, monotonically convergent
algorithms are obtained for a large class of quantum control problems. It
includes, in addition to non-linear equations of motion, control problems that
are characterized by non-unitary time evolution, non-linear dependencies of the
Hamiltonian on the control, time-dependent targets and optimization functionals
that depend to higher than second order on the time-evolving states. We
furthermore show that the non-linear (second order) contribution can be
estimated either analytically or numerically, yielding readily applicable
optimization algorithms. We demonstrate monotonic convergence for an
optimization functional that is an eighth-degree polynomial in the states. For
the 'standard' quantum control problem of a convex final-time functional,
linear equations of motion and linear dependency of the Hamiltonian on the
field, the second-order contribution is not required for monotonic convergence
but can be used to speed up convergence. We demonstrate this by comparing the
performance of first and second order algorithms for two examples.

本文证明基于 Konnov-Krotov 算法可获得一类大量量子控制问题的单调收敛算法，包括非线性运动方程、非幺正时间演化、Hamilton 算符的非线性依赖、时变目标以及高于时间演化状态二次的优化泛函。同时表明，这些问题中的二次贡献可通过解析或数值方法估算，产生可应用的优化算法。