This paper establishes a mathematical foundation for the Adam optimizer, elucidating its connection to natural gradient descent through Riemannian and information geometry. We rigorously analyze the diagonal empirical Fisher information matrix (FIM) in Adam, clarifying all detailed approximations and advocating for the use of log probability functions as loss, which should be based on discrete distributions, due to the limitations of empirical FIM. Our analysis uncovers flaws in the original Adam algorithm, leading to proposed corrections such as enhanced momentum calculations, adjusted bias corrections, and gradient clipping. We refine the weight decay term based on our theoretical framework. Our modified algorithm, Fisher Adam (FAdam), demonstrates superior performance across diverse domains including LLM, ASR, and VQ-VAE, achieving state-of-the-art results in ASR.

本文通过黎曼几何和信息几何，为Adam优化器建立了数学基础，阐明了其与自然梯度下降的联系。我们严格分析了Adam中的对角经验Fisher信息矩阵（FIM），明确了所有详细的近似方法，并主张使用基于离散分布的对数概率函数作为损失函数，这是由于经验FIM的局限性。我们的分析揭示了原始Adam算法的缺陷，并提出了修正方法，如增强动量计算、调整偏差校正和梯度修剪。我们根据理论框架改进了权重衰减项。我们修改后的算法Fisher Adam（FAdam）在包括LLM、ASR和VQ-VAE在内的不同领域展示了卓越的性能，并在ASR中取得了最先进的结果。

FAdam：Adam 是一种使用对角化经验费舍尔信息的自然梯度优化器