In recent years, pretrained neural language models (PNLMs) have taken the
field of natural language processing by storm, achieving new benchmarks and
state-of-the-art performances. These models often rely heavily on annotated
data, which may not always be available. Data scarcity are commonly found in
specialized domains, such as medical, or in low-resource languages that are
underexplored by AI research. In this dissertation, we focus on mitigating data
scarcity using data augmentation and neural ensemble learning techniques for
neural language models. In both research directions, we implement neural
network algorithms and evaluate their impact on assisting neural language
models in downstream NLP tasks. Specifically, for data augmentation, we explore
two techniques: 1) creating positive training data by moving an answer span
around its original context and 2) using text simplification techniques to
introduce a variety of writing styles to the original training data. Our
results indicate that these simple and effective solutions improve the
performance of neural language models considerably in low-resource NLP domains
and tasks. For neural ensemble learning, we use a multilabel neural classifier
to select the best prediction outcome from a variety of individual pretrained
neural language models trained for a low-resource medical text simplification
task.

本文研究了使用数据增强和神经集成学习技术来减轻预训练神经语言模型在医学或低资源语言等特定领域中的数据稀缺问题。实验结果表明，这些简单有效的解决方案可以显著提高这些领域下神经语言模型的性能。