Image captioning, like many tasks involving vision and language, currently
relies on Transformer-based architectures for extracting the semantics in an
image and translating it into linguistically coherent descriptions. Although
successful, the attention operator only considers a weighted summation of
projections of the current input sample, therefore ignoring the relevant
semantic information which can come from the joint observation of other
samples. In this paper, we devise a network which can perform attention over
activations obtained while processing other training samples, through a
prototypical memory model. Our memory models the distribution of past keys and
values through the definition of prototype vectors which are both
discriminative and compact. Experimentally, we assess the performance of the
proposed model on the COCO dataset, in comparison with carefully designed
baselines and state-of-the-art approaches, and by investigating the role of
each of the proposed components. We demonstrate that our proposal can increase
the performance of an encoder-decoder Transformer by 3.7 CIDEr points both when
training in cross-entropy only and when fine-tuning with self-critical sequence
training. Source code and trained models are available at:
this https URL

本文介绍了一种通过原型记忆模型，在处理其他训练样本时，可以执行对激活进行注意力操作的网络，并通过 COCO 数据集上的实验进行了评估。

借助过去的一点点帮助：用于图像字幕的原型记忆网络

With a Little Help from your own Past: Prototypical Memory Networks for  Image Captioning

Recent advances in deep learning have relied heavily on the use of large
Transformers due to their ability to learn at scale. However, the core building
block of Transformers, the attention operator, exhibits quadratic cost in
sequence length, limiting the amount of context accessible. Existing
subquadratic methods based on low-rank and sparse approximations need to be
combined with dense attention layers to match Transformers, indicating a gap in
capability. In this work, we propose Hyena, a subquadratic drop-in replacement
for attention constructed by interleaving implicitly parametrized long
convolutions and data-controlled gating. In recall and reasoning tasks on
sequences of thousands to hundreds of thousands of tokens, Hyena improves
accuracy by more than 50 points over operators relying on state-spaces and
other implicit and explicit methods, matching attention-based models. We set a
new state-of-the-art for dense-attention-free architectures on language
modeling in standard datasets (WikiText103 and The Pile), reaching Transformer
quality with a 20% reduction in training compute required at sequence length
2K. Hyena operators are twice as fast as highly optimized attention at sequence
length 8K, and 100x faster at sequence length 64K.

本文介绍了 Hyena，一种亚二次的 attention 替代方法，它通过交错的启发式参数化长卷积和数据控制开关的方式进行构建，能够有效地解决 Transformers 中 quadratic cost 的问题，并且在大规模自然语言处理任务中实现了 Transformer 模型一致的精度，同时减少了 20% 的训练计算资源。