A Lite BERT (ALBERT) has been introduced to scale up deep bidirectional
representation learning for natural languages. Due to the lack of pretrained
ALBERT models for Korean language, the best available practice is the
multilingual model or resorting back to the any other BERT-based model. In this
paper, we develop and pretrain KoreALBERT, a monolingual ALBERT model
specifically for Korean language understanding. We introduce a new training
objective, namely Word Order Prediction (WOP), and use alongside the existing
MLM and SOP criteria to the same architecture and model parameters. Despite
having significantly fewer model parameters (thus, quicker to train), our
pretrained KoreALBERT outperforms its BERT counterpart on 6 different NLU
tasks. Consistent with the empirical results in English by Lan et al.,
KoreALBERT seems to improve downstream task performance involving
multi-sentence encoding for Korean language. The pretrained KoreALBERT is
publicly available to encourage research and application development for Korean
NLP.

该研究开发并预训练了 KoreALBERT 模型，实现了针对韩语理解的单语言 ALBERT 模型，采用新的训练目标 - WOP，与现有的 MLM 和 SOP 方法结合，尽管参数较少，但预训练的 KoreALBERT 在 6 项不同的 NLU 任务中表现优异，并提升了韩语多句子编码相关的下游任务表现。

KoreALBERT：针对韩语语言理解的轻量级 BERT 模型预训练

KoreALBERT: Pretraining a Lite BERT Model for Korean Language  Understanding

We show how to predict the basic word-order facts of a novel language given
only a corpus of part-of-speech (POS) sequences. We predict how often direct
objects follow their verbs, how often adjectives follow their nouns, and in
general the directionalities of all dependency relations. Such typological
properties could be helpful in grammar induction. While such a problem is
usually regarded as unsupervised learning, our innovation is to treat it as
supervised learning, using a large collection of realistic synthetic languages
as training data. The supervised learner must identify surface features of a
language's POS sequence (hand-engineered or neural features) that correlate
with the language's deeper structure (latent trees). In the experiment, we
show: 1) Given a small set of real languages, it helps to add many synthetic
languages to the training data. 2) Our system is robust even when the POS
sequences include noise. 3) Our system on this task outperforms a grammar
induction baseline by a large margin.

本研究展示了如何通过部分语音序列对一个新语言的基本语序事实进行预测，并且通过大量的合成语言训练数据，将这个通常被认为是无监督学习的问题转化成了有监督学习，从而成功地识别了一个语言的 POS 序列的表面特征（手工特征或神经特征）与其更深层次的结构（潜在树）相关性。该系统在该任务上表现出色，比语法归纳基线高出一个很大的优势。