In this era of large language models (LLMs), the traditional training of
models has become increasingly unimaginable for regular users and institutions.
The exploration of efficient fine-tuning for high-resource languages on these
models is an undeniable trend that is gradually gaining popularity. However,
there has been very little exploration for various low-resource languages, such
as Tibetan. Research in Tibetan NLP is inherently scarce and limited. While
there is currently no existing large language model for Tibetan due to its
low-resource nature, that day will undoubtedly arrive. Therefore, research on
efficient fine-tuning for low-resource language models like Tibetan is highly
necessary. Our research can serve as a reference to fill this crucial gap.
Efficient fine-tuning strategies for pre-trained language models (PLMs) in
Tibetan have seen minimal exploration. We conducted three types of efficient
fine-tuning experiments on the publicly available TNCC-title dataset:
"prompt-tuning," "Adapter lightweight fine-tuning," and "prompt-tuning +
Adapter fine-tuning." The experimental results demonstrate significant
improvements using these methods, providing valuable insights for advancing
Tibetan language applications in the context of pre-trained models.

我们进行了针对藏语（一种低资源语言）的有效微调策略研究，包括信息提示微调、轻量级适配器微调和信息提示微调 + 适配器微调，实验证明这些方法在预训练模型中显著提高了效果，为提升藏语应用提供了有价值的见解。

PEFTT: 低资源藏语预训练语言模型的参数高效微调

PEFTT: Parameter-Efficient Fine-Tuning for low-resource Tibetan  pre-trained language models

The pre-trained language model is trained on large-scale unlabeled text and
can achieve state-of-the-art results in many different downstream tasks.
However, the current pre-trained language model is mainly concentrated in the
Chinese and English fields. For low resource language such as Tibetan, there is
lack of a monolingual pre-trained model. To promote the development of Tibetan
natural language processing tasks, this paper collects the large-scale training
data from Tibetan websites and constructs a vocabulary that can cover 99.95$\%$
of the words in the corpus by using Sentencepiece. Then, we train the Tibetan
monolingual pre-trained language model named TiBERT on the data and vocabulary.
Finally, we apply TiBERT to the downstream tasks of text classification and
question generation, and compare it with classic models and multilingual
pre-trained models, the experimental results show that TiBERT can achieve the
best performance. Our model is published in this http URL

本研究使用 Sentencepiece 从藏文网站收集大规模训练数据，并构建了一个词汇表，覆盖了语料库中 99.95％的单词，使用这些数据和词汇表训练了一个称为 TiBERT 的藏文单语言预训练语言模型，最后将 TiBERT 应用于文本分类和问题生成的下游任务，并且与传统模型和多语言预训练模型进行比较，实验结果表明 TiBERT 能够实现最佳性能。