As deep learning models continue to increase in size, the memory requirements
for training have surged. While high-level techniques like offloading,
recomputation, and compression can alleviate memory pressure, they also
introduce overheads. However, a memory-efficient execution plan that includes a
reasonable operator execution order and tensor memory layout can significantly
increase the models' memory efficiency and reduce overheads from high-level
techniques. In this paper, we propose ROAM which operates on computation graph
level to derive memory-efficient execution plan with optimized operator order
and tensor memory layout for models. We first propose sophisticated theories
that carefully consider model structure and training memory load to support
optimization for large complex graphs that have not been well supported in the
past. An efficient tree-based algorithm is further proposed to search task
divisions automatically, along with delivering high performance and
effectiveness to solve the problem. Experiments show that ROAM achieves a
substantial memory reduction of 35.7%, 13.3%, and 27.2% compared to Pytorch and
two state-of-the-art methods and offers a remarkable 53.7x speedup. The
evaluation conducted on the expansive GPT2-XL further validates ROAM's
scalability.

ROAM 提出了一种在计算图级别上操作，以得到内存有效的执行计划，包括优化的运算符顺序和张量内存布局，用于模型。ROAM 相对于 Pytorch 和两种最新方法能够显著降低 35.7%、13.3% 和 27.2% 的内存需求并达到了惊人的 53.7 倍加速。