Training large-scale machine learning models poses distinct system
challenges, given both the size and complexity of today's workloads. Recently,
many organizations training state-of-the-art Generative AI models have reported
cases of instability during training, often taking the form of loss spikes.
Numeric deviation has emerged as a potential cause of this training
instability, although quantifying this is especially challenging given the
costly nature of training runs. In this work, we develop a principled approach
to understanding the effects of numeric deviation, and construct proxies to put
observations into context when downstream effects are difficult to quantify. As
a case study, we apply this framework to analyze the widely-adopted Flash
Attention optimization. We find that Flash Attention sees roughly an order of
magnitude more numeric deviation as compared to Baseline Attention at BF16 when
measured during an isolated forward pass. We then use a data-driven analysis
based on the Wasserstein Distance to provide upper bounds on how this numeric
deviation impacts model weights during training, finding that the numerical
deviation present in Flash Attention is 2-5 times less significant than
low-precision training.

利用数值偏差原则性地分析了大规模机器学习模型训练中的稳定性问题，以 Flash Attention 优化为研究对象，发现其数值偏差比低精度训练要小 2-5 倍。

闪光关注是否稳定？

Is Flash Attention Stable?

Recent developments in large-scale machine learning models for
general-purpose understanding, translation and generation of language are
driving impact across a variety of sectors including medicine, robotics, and
scientific discovery. The strength of such Large Language Models (LLMs) stems
from the large corpora that they are trained with. While this imbues them with
a breadth of capabilities, they have been found unsuitable for some specific
types of problems such as advanced mathematics. In this paper, we highlight the
inability of LLMs to reason about physics tasks. We demonstrate that their
ability to infer parameters of physical systems can be improved, without
retraining, by augmenting their context with feedback from physical simulation.

大规模机器学习模型在语言的通用理解、翻译和生成方面的最新进展，给医学、机器人学和科学探索等多个领域带来了影响。然而，这些大语言模型因为其训练所使用的大型语料库的特点，存在无法处理一些特定问题，如高级数学。本文针对大语言模型无法进行物理任务推理的问题进行了研究，并证明了通过从物理模拟的反馈中增强它们的上下文，可以改进它们推断物理系统参数的能力，而无需重新训练。