Transformer-based models have achieved great success in various NLP, vision,
and speech tasks. However, the core of Transformer, the self-attention
mechanism, has a quadratic time and memory complexity with respect to the
sequence length, which hinders applications of Transformer-based models to long
sequences. Many approaches have been proposed to mitigate this problem, such as
sparse attention mechanisms, low-rank matrix approximations and scalable
kernels, and token mixing alternatives to self-attention. We propose a novel
Pooling Network (PoNet) for token mixing in long sequences with linear
complexity. We design multi-granularity pooling and pooling fusion to capture
different levels of contextual information and combine their interactions with
tokens. On the Long Range Arena benchmark, PoNet significantly outperforms
Transformer and achieves competitive accuracy, while being only slightly slower
than the fastest model, FNet, across all sequence lengths measured on GPUs. We
also conduct systematic studies on the transfer learning capability of PoNet
and observe that PoNet achieves 95.7% of the accuracy of BERT on the GLUE
benchmark, outperforming FNet by 4.5% relative. Comprehensive ablation analysis
demonstrates effectiveness of the designed multi-granularity pooling and
pooling fusion for token mixing in long sequences and efficacy of the designed
pre-training tasks for PoNet to learn transferable contextualized language
representations.

本文提出了一种新的 Pooling Network (PoNet)，它使用线性复杂度的 token mixing 处理长序列，通过多粒度池化和池化融合来捕获不同级别的上下文信息，并结合 tokens 进行交互来提高模型性能。在长范围竞技基准测试中，PoNet 显著优于 Transformer，同时实现与最快模型 FNet 相当的准确性，在 GPU 上度量所有序列长度时仅比最快模型慢一点。同时文章也进行了系统的研究，证明了 PoNet 设计的多粒度池化和池化融合的加强长序列的 token 混合以及设计的预训练任务可用于学习可转移的上下文化语言表示的有效性。

PoNet: 长序列中高效的令牌混合池化网络

PoNet: Pooling Network for Efficient Token Mixing in Long Sequences

A key problem in blind image quality assessment (BIQA) is how to effectively
model the properties of human visual system in a data-driven manner. In this
paper, we propose a simple and efficient BIQA model based on a novel framework
which consists of a fully convolutional neural network (FCNN) and a pooling
network to solve this problem. In principle, FCNN is capable of predicting a
pixel-by-pixel similar quality map only from a distorted image by using the
intermediate similarity maps derived from conventional full-reference image
quality assessment methods. The predicted pixel-by-pixel quality maps have good
consistency with the distortion correlations between the reference and
distorted images. Finally, a deep pooling network regresses the quality map
into a score. Experiments have demonstrated that our predictions outperform
many state-of-the-art BIQA methods.

本文介绍了一种基于卷积神经网络和池化网络的盲图像质量评估模型，能够有效地模拟人类视觉系统的特性，并通过全参考图像质量评估方法获得的中间相似度图来预测从扭曲的图像中像素之间的质量。最终结果通过深池化网络回归为得分，该方法表现优于许多现有的盲图像质量评估方法。