Runtime and scalability of large neural networks can be significantly
affected by the placement of operations in their dataflow graphs on suitable
devices. With increasingly complex neural network architectures and
heterogeneous device characteristics, finding a reasonable placement is
extremely challenging even for domain experts. Most existing automated device
placement approaches are impractical due to the significant amount of compute
required and their inability to generalize to new, previously held-out graphs.
To address both limitations, we propose an efficient end-to-end method based on
a scalable sequential attention mechanism over a graph neural network that is
transferable to new graphs. On a diverse set of representative deep learning
models, including Inception-v3, AmoebaNet, Transformer-XL, and WaveNet, our
method on average achieves 16% improvement over human experts and 9.2%
improvement over the prior art with 15 times faster convergence. To further
reduce the computation cost, we pre-train the policy network on a set of
dataflow graphs and use a superposition network to fine-tune it on each
individual graph, achieving state-of-the-art performance on large hold-out
graphs with over 50k nodes, such as an 8-layer GNMT.

本文提出了一种端到端的基于图神经网络的可扩展顺序关注机制，并使用超前网络在每个独立图上进行微调，达到了优于人类专家和先前技术的 16% 平均改善和 15 倍更快的收敛速度，从而实现了有关大型神经网络和设备放置的计算成本和效率的突破。