Real-world data is high-dimensional: a book, image, or musical performance
can easily contain hundreds of thousands of elements even after compression.
However, the most commonly used autoregressive models, Transformers, are
prohibitively expensive to scale to the number of inputs and layers needed to
capture this long-range structure. We develop Perceiver AR, an autoregressive,
modality-agnostic architecture which uses cross-attention to map long-range
inputs to a small number of latents while also maintaining end-to-end causal
masking. Perceiver AR can directly attend to over a hundred thousand tokens,
enabling practical long-context density estimation without the need for
hand-crafted sparsity patterns or memory mechanisms. When trained on images or
music, Perceiver AR generates outputs with clear long-term coherence and
structure. Our architecture also obtains state-of-the-art likelihood on
long-sequence benchmarks, including 64 x 64 ImageNet images and PG-19 books.

使用跨注意力机制构建了 Perceiver AR 机器学习模型，可自动学习高维的图像和音乐数据中的长期依赖关系，并在图像和书籍的测试中取得最优的结果，而无需人为设计稀疏模式或记忆机制。

Perceiver AR 实现的通用、长上下文自回归建模

General-purpose, long-context autoregressive modeling with Perceiver AR

Gatys et al. (2015) showed that pair-wise products of features in a
convolutional network are a very effective representation of image textures. We
propose a simple modification to that representation which makes it possible to
incorporate long-range structure into image generation, and to render images
that satisfy various symmetry constraints. We show how this can greatly improve
rendering of regular textures and of images that contain other kinds of
symmetric structure. We also present applications to inpainting and season
transfer.

本文提出一种对卷积网络的特征进行简单修改的方法，可以将长程结构纳入图像生成，并渲染满足各种对称约束的图像以及在图像修复和季节转换等方面应用广泛。