Recent works have highlighted the strength of the Transformer architecture on
sequence tasks while, at the same time, neural architecture search (NAS) has
begun to outperform human-designed models. Our goal is to apply NAS to search
for a better alternative to the Transformer. We first construct a large search
space inspired by the recent advances in feed-forward sequence models and then
run evolutionary architecture search with warm starting by seeding our initial
population with the Transformer. To directly search on the computationally
expensive WMT 2014 English-German translation task, we develop the Progressive
Dynamic Hurdles method, which allows us to dynamically allocate more resources
to more promising candidate models. The architecture found in our experiments
-- the Evolved Transformer -- demonstrates consistent improvement over the
Transformer on four well-established language tasks: WMT 2014 English-German,
WMT 2014 English-French, WMT 2014 English-Czech and LM1B. At a big model size,
the Evolved Transformer establishes a new state-of-the-art BLEU score of 29.8
on WMT'14 English-German; at smaller sizes, it achieves the same quality as the
original "big" Transformer with 37.6% less parameters and outperforms the
Transformer by 0.7 BLEU at a mobile-friendly model size of 7M parameters.

本研究旨在使用神经架构搜索技术应用于 Transformer 结构中，通过进化架构搜索技术，探索发现了性能更好的替代模型 Evolved Transformer，该模型不仅在 WMT 2014 英德翻译任务上取得了新的性能最优结果，且在模型容量更小的情况下性能优于原始模型。