We present a new layer in which dynamic (i.e.,input-dependent) Infinite
Impulse Response (IIR) filters of order two are used to process the input
sequence prior to applying conventional attention. The input is split into
chunks, and the coefficients of these filters are determined based on previous
chunks to maintain causality. Despite their relatively low order, the causal
adaptive filters are shown to focus attention on the relevant sequence
elements. The layer performs on-par with state of the art networks, with a
fraction of the parameters and with time complexity that is sub-quadratic with
input size. The obtained layer is favorable to layers such as Heyna, GPT2, and
Mega, both with respect to the number of parameters and the obtained level of
performance on multiple long-range sequence problems.

本文提出了一种新的层，利用动态的、具有自适应功能的 IIR 滤波器对输入序列进行处理，从而实现与现有顶尖网络相媲美的性能，且参数更少、时间复杂度更低，适用于多个长序列问题。

使用自适应 IIR 滤波器集中注意力

Focus Your Attention (with Adaptive IIR Filters)

We propose a contrastive attention mechanism to extend the
sequence-to-sequence framework for abstractive sentence summarization task,
which aims to generate a brief summary of a given source sentence. The proposed
contrastive attention mechanism accommodates two categories of attention: one
is the conventional attention that attends to relevant parts of the source
sentence, the other is the opponent attention that attends to irrelevant or
less relevant parts of the source sentence. Both attentions are trained in an
opposite way so that the contribution from the conventional attention is
encouraged and the contribution from the opponent attention is discouraged
through a novel softmax and softmin functionality. Experiments on benchmark
datasets show that, the proposed contrastive attention mechanism is more
focused on the relevant parts for the summary than the conventional attention
mechanism, and greatly advances the state-of-the-art performance on the
abstractive sentence summarization task. We release the code at
this https URL

提出了一种对抽象句子摘要任务的序列到序列框架进行扩展的对比注意机制，以生成给定源句子的简要摘要，并在基准数据集上进行实验证明，在摘要中更关注源句子的相关部分比传统的注意机制更有效。