BERT has shown a lot of sucess in a wide variety of NLP tasks. But it has a
limitation dealing with long inputs due to its attention mechanism. Longformer,
ETC and BigBird addressed this issue and effectively solved the quadratic
dependency problem. However we find that these models are not sufficient, and
propose LittleBird, a novel model based on BigBird with improved speed and
memory footprint while maintaining accuracy. In particular, we devise a more
flexible and efficient position representation method based on Attention with
Linear Biases (ALiBi). We also show that replacing the method of global
information represented in the BigBird with pack and unpack attention is more
effective. The proposed model can work on long inputs even after being
pre-trained on short inputs, and can be trained efficiently reusing existing
pre-trained language model for short inputs. This is a significant benefit for
low-resource languages where large amounts of long text data are difficult to
obtain. As a result, our experiments show that LittleBird works very well in a
variety of languages, achieving high performance in question answering tasks,
particularly in KorQuAD2.0, Korean Question Answering Dataset for long
paragraphs.

提出了一个新模型 LittleBird，它是基于 BigBird 的改进版本，使用 Attention with Linear Biases（ALiBi）的更灵活、高效的位置表示方法，维护了准确性的同时，在速度和内存占用方面进行了优化。实验证明，LittleBird 在各种语言中表现良好，在特定的 KorQuAD2.0 数据集中，在长段落问答任务中取得了高精度