MEGA is a recent transformer-based architecture, which utilizes a linear
recurrent operator whose parallel computation, based on the FFT, scales as
$O(LlogL)$, with $L$ being the sequence length. We build upon their approach by
replacing the linear recurrence with a special temporal convolutional network
which permits larger receptive field size with shallower networks, and reduces
the computational complexity to $O(L)$. The resulting model is called TCNCA, a
Temporal Convolutional Network with Chunked Attention. We evaluate TCNCA on
EnWik8 language modeling, long-range-arena (LRA) sequence classification, as
well as a synthetic reasoning benchmark associative recall. On EnWik8, TCNCA
outperforms MEGA, reaching a lower loss with $1.37\times$/$1.24\times$ faster
forward/backward pass during training. The dilated convolutions used in TCNCA
are consistently and significantly faster operations than the FFT-based
parallelized recurrence in GPUs, making them a scalable candidate for handling
very large sequence lengths: they are up to $7.07\times$/$2.86\times$ faster in
the forward/backward pass for sequences up to 131k. Further on LRA, TCNCA
achieves, on average, $1.28\times$ speed-up during inference with similar
accuracy to what MEGA achieves. On associative recall, we find that even a
simplified version of TCNCA, without excessive multiplicative and additive
interactions, remains superior or competitive to MEGA on a range of sequence
lengths and vocabulary sizes.

TCNCA 是一种具有切分关注力的时间卷积网络，通过用一种特殊的时间卷积网络替换线性递归操作，在减少计算复杂度的同时提供更大的感受野大小和更浅的网络结构。在多个任务中，TCNCA 表现优于 MEGA，并在处理非常长的序列长度时具有可扩展性。