With the development of feed-forward models, the default model for sequence
modeling has gradually evolved to replace recurrent networks. Many powerful
feed-forward models based on convolutional networks and attention mechanism
were proposed and show more potential to handle sequence modeling tasks. We
wonder that is there an architecture that can not only achieve an approximate
substitution of recurrent network, but also absorb the advantages of
feed-forward models. So we propose an exploratory architecture referred to
Temporal Convolutional Attention-based Network (TCAN) which combines temporal
convolutional network and attention mechanism. TCAN includes two parts, one is
Temporal Attention (TA) which captures relevant features inside the sequence,
the other is Enhanced Residual (ER) which extracts shallow layer's important
information and transfers to deep layers. We improve the state-of-the-art
results of bpc/perplexity to 26.92 on word-level PTB, 1.043 on character-level
PTB, and 6.66 on WikiText-2.

我们提出了一种基于时间卷积网络和注意力机制的探索性架构称为 TCAN，它不仅能够实现递归网络的近似替代，还可以吸收前向模型的优势，提高了 word-level PTB、character-level PTB 和 WikiText-2 等文本数据集的 bpc/perplexity 表现.