The pre-trained language models have achieved great successes in various
natural language understanding (NLU) tasks due to its capacity to capture the
deep contextualized information in text by pre-training on large-scale corpora.
One of the fundamental components in pre-trained language models is the
vocabulary, especially for training multilingual models on many different
languages. In the technical report, we present our practices on training
multilingual pre-trained language models with BBPE: Byte-Level BPE (i.e., Byte
Pair Encoding). In the experiment, we adopted the architecture of NEZHA as the
underlying pre-trained language model and the results show that NEZHA trained
with byte-level subwords consistently outperforms Google multilingual BERT and
vanilla NEZHA by a notable margin in several multilingual NLU tasks. We release
the source code of our byte-level vocabulary building tools and the
multilingual pre-trained language models.

本文介绍了使用 Byte-Level BPE 技术训练 NEZHA 多语言预训练语言模型，并通过多语言 NLU 任务验证其性能显着优于 Google 多语言 BERT 和原始 NEZHA 模型。

使用字节级子词训练多语言预训练语言模型

Training Multilingual Pre-trained Language Model with Byte-level  Subwords

The pre-trained language models have achieved great successes in various
natural language understanding (NLU) tasks due to its capacity to capture the
deep contextualized information in text by pre-training on large-scale corpora.
In this technical report, we present our practice of pre-training language
models named NEZHA (NEural contextualiZed representation for CHinese lAnguage
understanding) on Chinese corpora and finetuning for the Chinese NLU tasks. The
current version of NEZHA is based on BERT with a collection of proven
improvements, which include Functional Relative Positional Encoding as an
effective positional encoding scheme, Whole Word Masking strategy, Mixed
Precision Training and the LAMB Optimizer in training the models. The
experimental results show that NEZHA achieves the state-of-the-art performances
when finetuned on several representative Chinese tasks, including named entity
recognition (People's Daily NER), sentence matching (LCQMC), Chinese sentiment
classification (ChnSenti) and natural language inference (XNLI).

NEZHA 是对中文语言理解进行预训练的语言模型，基于 BERT，并经过一系列优化策略，包括 Functional Relative Positional Encoding 编码方案、Whole Word Masking 策略、Mixed Precision Training 和 LAMB Optimizer 等，实验证明 NEZHA 在多个中文 NLU 任务上达到了最先进的性能。