In this paper, we present the Fast Optimizer Benchmark (FOB), a tool designed
for evaluating deep learning optimizers during their development. The benchmark
supports tasks from multiple domains such as computer vision, natural language
processing, and graph learning. The focus is on convenient usage, featuring
human-readable YAML configurations, SLURM integration, and plotting utilities.
FOB can be used together with existing hyperparameter optimization (HPO) tools
as it handles training and resuming of runs. The modular design enables
integration into custom pipelines, using it simply as a collection of tasks. We
showcase an optimizer comparison as a usage example of our tool. FOB can be
found on GitHub: this https URL

我们介绍了快速优化器基准（FOB），这是一种用于在其开发过程中评估深度学习优化器的工具。该基准支持来自多个领域（如计算机视觉、自然语言处理和图学习）的任务。其重点在于方便使用，具有人类可读的 YAML 配置、SLURM 集成和绘图实用程序。FOB 可与现有的超参数优化（HPO）工具一起使用，因为它处理训练和恢复运行。模块化设计使其能够集成到自定义流水线中，只需将其作为任务集合使用。我们通过一个优化器比较的示例展示了我们的工具的用法。FOB 可以在 GitHub 上找到：https://github.com/。

快速优化器基准测试

Fast Optimizer Benchmark

Recently, flat-minima optimizers, which seek to find parameters in low-loss
neighborhoods, have been shown to improve a neural network's generalization
performance over stochastic and adaptive gradient-based optimizers. Two methods
have received significant attention due to their scalability: 1. Stochastic
Weight Averaging (SWA), and 2. Sharpness-Aware Minimization (SAM). However,
there has been limited investigation into their properties and no systematic
benchmarking of them across different domains. We fill this gap here by
comparing the loss surfaces of the models trained with each method and through
broad benchmarking across computer vision, natural language processing, and
graph representation learning tasks. We discover several surprising findings
from these results, which we hope will help researchers further improve deep
learning optimizers, and practitioners identify the right optimizer for their
problem.

通过比较基于平坦极小点优化器的损失曲面和在计算机视觉、自然语言处理和图表示学习任务的广泛基准测试中，我们发现了一些令人惊讶的发现，希望这能帮助研究人员进一步改进深度学习优化器，并帮助实践者为其问题选择正确的优化器。