We formalize the problem of trading-off DNN training time and memory
requirements as the tensor rematerialization optimization problem, a
generalization of prior checkpointing strategies. We introduce Checkmate, a
system that solves for optimal rematerialization schedules in reasonable times
(under an hour) using off-the-shelf MILP solvers or near-optimal schedules with
an approximation algorithm, then uses these schedules to accelerate millions of
training iterations. Our method scales to complex, realistic architectures and
is hardware-aware through the use of accelerator-specific, profile-based cost
models. In addition to reducing training cost, Checkmate enables real-world
networks to be trained with up to 5.1x larger input sizes. Checkmate is an
open-source project, available at this https URL

本篇研究论文提出了 Checkmate，这是一种针对训练 DNN 进行时间和内存要求的权衡的系统，它可以在不到一小时的时间内解决最佳重构日程表，并使用这些日程表加速数百万次的训练迭代，可适用于复杂的网络结构。