This work presents KoBigBird-large, a large size of Korean BigBird that
achieves state-of-the-art performance and allows long sequence processing for
Korean language understanding. Without further pretraining, we only transform
the architecture and extend the positional encoding with our proposed Tapered
Absolute Positional Encoding Representations (TAPER). In experiments,
KoBigBird-large shows state-of-the-art overall performance on Korean language
understanding benchmarks and the best performance on document classification
and question answering tasks for longer sequences against the competitive
baseline models. We publicly release our model here.

本研究介绍了一种名为 KoBigBird-large 的大规模韩语 BigBird 模型，它能够在韩语语言理解方面实现领先水平并处理长序列。通过采用作者提出的锥形绝对位置编码表示（TAPER），无需进一步预训练，只需改变架构和扩展位置编码。实验证明，KoBigBird-large 在韩语语言理解基准测试中表现出类似的领先水平，并在长序列上的文档分类和问题回答任务中展现了与竞争基准模型相比更好的性能。本研究在此公开发布了我们的模型。