We present the design of a new large scale orchestration layer for
accelerators. Our system, Pathways, is explicitly designed to enable
exploration of new systems and ML research ideas, while retaining state of the
art performance for current models. Pathways uses a sharded dataflow graph of
asynchronous operators that consume and produce futures, and efficiently
gang-schedules heterogeneous parallel computations on thousands of accelerators
while coordinating data transfers over their dedicated interconnects. Pathways
makes use of a novel asynchronous distributed dataflow design that lets the
control plane execute in parallel despite dependencies in the data plane. This
design, with careful engineering, allows Pathways to adopt a single-controller
model that makes it easier to express complex new parallelism patterns. We
demonstrate that Pathways can achieve performance parity (~100% accelerator
utilization) with state-of-the-art systems when running SPMD computations over
2048 TPUs, while also delivering throughput comparable to the SPMD case for
Transformer models that are pipelined across 16 stages, or sharded across two
islands of accelerators connected over a data center network.

我们设计了一种新型大规模编排层，Pathways，它能够探索新的系统和机器学习研究想法，同时保持目前模型的先进性能。Pathways 使用异步 operator 的分片数据流图，在数千个加速器上高效地安排异构并行计算，同时通过专用互连协调数据传输。通过小心的工程技术，Pathways 采用了异步分布式数据流设计，使得控制平面能够并行执行，尽管在数据平面有依赖。我们证明，Pathways 能够在 2048 TPUs 上运行 SPMD 计算，达到与最先进系统相当的性能平衡（~100% 加速器利用率），同时在 Transformer 模型的 16 个阶段或连接在数据中心网络上的两个加速器岛上进行分片时，也能够提供与 SPMD 情况相当的吞吐量。

路径：面向机器学习的异步分布式数据流

Pathways: Asynchronous Distributed Dataflow for ML

We describe an approach to task-oriented dialogue in which dialogue state is
represented as a dataflow graph. A dialogue agent maps each user utterance to a
program that extends this graph. Programs include metacomputation operators for
reference and revision that reuse dataflow fragments from previous turns. Our
graph-based state enables the expression and manipulation of complex user
intents, and explicit metacomputation makes these intents easier for learned
models to predict. We introduce a new dataset, SMCalFlow, featuring complex
dialogues about events, weather, places, and people. Experiments show that
dataflow graphs and metacomputation substantially improve representability and
predictability in these natural dialogues. Additional experiments on the
MultiWOZ dataset show that our dataflow representation enables an otherwise
off-the-shelf sequence-to-sequence model to match the best existing
task-specific state tracking model. The SMCalFlow dataset and code for
replicating experiments are available at
this https URL

本研究提出了一种基于数据流图表示对话状态并引入元计算符以复用过去状态的方法，并针对此开发了 SMCalFlow 数据集以进行有关事件、天气、地点和人物等问题的复杂对话，实验证明基于数据流图和元计算的方法可显著提高此类自然对话的可表达性和可预测性，同时将其与现有的任务领域状态跟踪模型进行了比较。

面向任务的对话作为数据流综合

Task-Oriented Dialogue as Dataflow Synthesis

This paper presents Tofu, a system that partitions very large DNN models
across multiple GPU devices to reduce per-GPU memory footprint. Tofu is
designed to partition a dataflow graph of fine-grained tensor operators in
order to work transparently with a general-purpose deep learning platform like
MXNet. In order to automatically partition each operator, we propose to
describe the semantics of an operator in a simple language which represents
tensors as lambda functions mapping from tensor coordinates to values. To
optimally partition different operators in a dataflow graph, Tofu uses a
recursive search algorithm that minimizes the total communication cost. Our
experiments on an 8-GPU machine show that Tofu enables the training of very
large CNN and RNN models. It also achieves 25% - 400% speedup over alternative
approaches to train very large models.

Tofu 是一个系统，它可以将超大的 DNN 模型分割到多个 GPU 设备上以减少每个 GPU 的内存占用，该系统采用数据流图的方式透明地与 MXNet 等通用深度学习平台配合使用，通过一种描述算子语义的简单语言，实现自动划分。通过递归搜索算法来优化划分，并在 8-GPU 机器上的实验中展现了训练超大 CNN 和 RNN 模型的能力，在速度方面则比其他方法提高了 25% 至 400%。