Large language models (LLMs) have demonstrated exceptional performance in
various NLP applications. However, the majority of existing open-source LLMs
are pre-trained primarily on English data and little part of other languages.
This deficiency in multilingual training data results in suboptimal performance
when applied to languages with fewer available resources. Furthermore,
enhancing the performance of LLMs on low-resource languages by full-parameter
fine-tuning with additional data requires substantial computational resources,
posing computational barriers for research organizations and individual
researchers. Consequently, several techniques such as parameter-efficient
tuning and advanced embedding initialization have been proposed to address
these challenges. In this work, we combine them to facilitate cross-lingual
transfer on English-dominated open-source LLM. To effectively enhance the
model's proficiency in Traditional Chinese, we conduct secondary pre-training
on Llama 2 7B with Traditional Chinese data by leveraging QLoRA and our
proposed zip-tie embedding initialization. The resulting model called Bailong,
which stands for Bilingual trAnsfer learnIng based on qLOra and zip-tie
embeddiNG. We present Bailong-instruct 7B, a fine-tuned version of Bailong 7B
optimized for multi-turn dialogue scenarios. Recognizing the inadequacy of
benchmark datasets in Traditional Chinese, we further introduce Bailong-bench
to assess the alignment of models with human preferences and the capability to
follow instructions in both Traditional Chinese and English tasks. In our
evaluation, Bailong-instruct 7B exhibits competitive performance on
Bailong-bench and other benchmark datasets when compared to other open-source
models of similar or even larger parameter sizes. Bailong-instruct 7B and
Bailong-bench are publicly available with the aim of empowering the community
to build upon our efforts.

我们结合参数高效调整和高级嵌入初始化技术，通过对英语主导的开源大语言模型进行二次预训练，以提高其在繁体中文上的表现，并进一步引入评估平台来检验模型与人类偏好和指令遵循能力的一致性。我们的评估结果显示，我们的模型在多轮对话任务中相比其他类似的开源模型表现出竞争性水平，并提供了 Bailong-instruct 7B 与 Bailong-bench 供社区使用。

Bailong：基于 QLoRA 和 Zip-tie 嵌入的双语迁移学习

Bailong: Bilingual Transfer Learning based on QLoRA and Zip-tie  Embedding

Studies in bias and fairness in natural language processing have primarily
examined social biases within a single language and/or across few attributes
(e.g. gender, race). However, biases can manifest differently across various
languages for individual attributes. As a result, it is critical to examine
biases within each language and attribute. Of equal importance is to study how
these biases compare across languages and how the biases are affected when
training a model on multilingual data versus monolingual data. We present a
bias analysis across Italian, Chinese, English, Hebrew, and Spanish on the
downstream sentiment analysis task to observe whether specific demographics are
viewed more positively. We study bias similarities and differences across these
languages and investigate the impact of multilingual vs. monolingual training
data. We adapt existing sentiment bias templates in English to Italian,
Chinese, Hebrew, and Spanish for four attributes: race, religion, nationality,
and gender. Our results reveal similarities in bias expression such as
favoritism of groups that are dominant in each language's culture (e.g.
majority religions and nationalities). Additionally, we find an increased
variation in predictions across protected groups, indicating bias
amplification, after multilingual finetuning in comparison to multilingual
pretraining.

本论文对多种语言进行偏见分析，研究多语言数据训练与单语言数据训练对 Bias 的影响并使用情感分析模型来观察不同民族是否被更正面地看待，结果发现多语言 Finetuning 导致对保护群体的预测变异性增加，表明了 Bias 的放大效应。

比较多语言训练的偏差及其影响

Comparing Biases and the Impact of Multilingual Training across Multiple  Languages

One of the first steps in the utterance interpretation pipeline of many
task-oriented conversational AI systems is to identify user intents and the
corresponding slots. Since data collection for machine learning models for this
task is time-consuming, it is desirable to make use of existing data in a
high-resource language to train models in low-resource languages. However,
development of such models has largely been hindered by the lack of
multilingual training data. In this paper, we present a new data set of 57k
annotated utterances in English (43k), Spanish (8.6k) and Thai (5k) across the
domains weather, alarm, and reminder. We use this data set to evaluate three
different cross-lingual transfer methods: (1) translating the training data,
(2) using cross-lingual pre-trained embeddings, and (3) a novel method of using
a multilingual machine translation encoder as contextual word representations.
We find that given several hundred training examples in the the target
language, the latter two methods outperform translating the training data.
Further, in very low-resource settings, multilingual contextual word
representations give better results than using cross-lingual static embeddings.
We also compare the cross-lingual methods to using monolingual resources in the
form of contextual ELMo representations and find that given just small amounts
of target language data, this method outperforms all cross-lingual methods,
which highlights the need for more sophisticated cross-lingual methods.

本文讨论了在任务导向的对话系统中如何识别用户意图和相应的插槽，并通过多种跨语言转移方法进行数据集训练比较