Attention-free language models that combine gating and convolutions are
growing in popularity due to their efficiency and increasingly competitive
performance. To better understand these architectures, we pretrain a suite of
17 attention and "gated-convolution" language models, finding that SoTA
gated-convolution architectures still underperform attention by up to 2.1
perplexity points on the Pile. In fine-grained analysis, we find 82% of the gap
is explained by each model's ability to recall information that is previously
mentioned in-context, e.g. "Hakuna Matata means no worries Hakuna Matata it
means no" $\rightarrow$ "??". On this task, termed "associative recall", we
find that attention outperforms gated-convolutions by a large margin: a 70M
parameter attention model outperforms a 1.4 billion parameter gated-convolution
model on associative recall. This is surprising because prior work shows gated
convolutions can perfectly solve synthetic tests for AR capability. To close
the gap between synthetics and real language, we develop a new formalization of
the task called multi-query associative recall (MQAR) that better reflects
actual language. We perform an empirical and theoretical study of MQAR that
elucidates differences in the parameter-efficiency of attention and
gated-convolution recall. Informed by our analysis, we evaluate simple
convolution-attention hybrids and show that hybrids with input-dependent sparse
attention patterns can close 97.4% of the gap to attention, while maintaining
sub-quadratic scaling. Our code is accessible at:
this https URL

我们研究了无需注意力的语言模型，结合了门控和卷积的方法，发现在上下文角色联想回忆方面，注意力模型优于门控卷积模型。通过开发一项新的任务形式，多查询角色联想回忆（MQAR），我们证明了注意力和门控卷积模型之间参数效率的差异，并提出了输入相关的稀疏注意力模式来减小两者之间的差距，并且保持亚二次缩放。