This paper presents Tofu, a system that partitions very large DNN models
across multiple GPU devices to reduce per-GPU memory footprint. Tofu is
designed to partition a dataflow graph of fine-grained tensor operators in
order to work transparently with a general-purpose deep learning platform like
MXNet. In order to automatically partition each operator, we propose to
describe the semantics of an operator in a simple language which represents
tensors as lambda functions mapping from tensor coordinates to values. To
optimally partition different operators in a dataflow graph, Tofu uses a
recursive search algorithm that minimizes the total communication cost. Our
experiments on an 8-GPU machine show that Tofu enables the training of very
large CNN and RNN models. It also achieves 25% - 400% speedup over alternative
approaches to train very large models.

Tofu 是一个系统，它可以将超大的 DNN 模型分割到多个 GPU 设备上以减少每个 GPU 的内存占用，该系统采用数据流图的方式透明地与 MXNet 等通用深度学习平台配合使用，通过一种描述算子语义的简单语言，实现自动划分。通过递归搜索算法来优化划分，并在 8-GPU 机器上的实验中展现了训练超大 CNN 和 RNN 模型的能力，在速度方面则比其他方法提高了 25% 至 400%。