Dot-product attention has wide applications in computer vision and natural
language processing. However, its memory and computational costs grow
quadratically with the input size. Such growth prohibits its application on
high-resolution inputs. To remedy this drawback, this paper proposes a novel
efficient attention mechanism equivalent to dot-product attention but with
substantially less memory and computational costs. Its resource efficiency
allows more widespread and flexible integration of attention modules into a
network, which leads to better accuracies. Empirical evaluations demonstrated
the effectiveness of its advantages. Efficient attention modules brought
significant performance boosts to object detectors and instance segmenters on
MS-COCO 2017. Further, the resource efficiency democratizes attention to
complex models, where high costs prohibit the use of dot-product attention. As
an exemplar, a model with efficient attention achieved state-of-the-art
accuracies for stereo depth estimation on the Scene Flow dataset. Code is
available at this https URL

本文提出了一种新的有效注意机制，其在计算代价和内存使用方面远优于点积注意力，这种资源效率使得它能够更广泛和灵活地集成到网络中，提高了观察精度，可以用于物体检测、实例分割和立体深度估计等领域。