Current computational approaches for analysing or generating code-mixed
sentences do not explicitly model "naturalness" or "acceptability" of
code-mixed sentences, but rely on training corpora to reflect distribution of
acceptable code-mixed sentences. Modelling human judgement for the
acceptability of code-mixed text can help in distinguishing natural code-mixed
text and enable quality-controlled generation of code-mixed text. To this end,
we construct Cline - a dataset containing human acceptability judgements for
English-Hindi (en-hi) code-mixed text. Cline is the largest of its kind with
16,642 sentences, consisting of samples sourced from two sources: synthetically
generated code-mixed text and samples collected from online social media. Our
analysis establishes that popular code-mixing metrics such as CMI, Number of
Switch Points, Burstines, which are used to filter/curate/compare code-mixed
corpora have low correlation with human acceptability judgements, underlining
the necessity of our dataset. Experiments using Cline demonstrate that simple
Multilayer Perceptron (MLP) models trained solely on code-mixing metrics are
outperformed by fine-tuned pre-trained Multilingual Large Language Models
(MLLMs). Specifically, XLM-Roberta and Bernice outperform IndicBERT across
different configurations in challenging data settings. Comparison with
ChatGPT's zero and fewshot capabilities shows that MLLMs fine-tuned on larger
data outperform ChatGPT, providing scope for improvement in code-mixed tasks.
Zero-shot transfer from English-Hindi to English-Telugu acceptability judgments
using our model checkpoints proves superior to random baselines, enabling
application to other code-mixed language pairs and providing further avenues of
research. We publicly release our human-annotated dataset, trained checkpoints,
code-mix corpus, and code for data generation and model training.

当前分析或生成混合代码句子的计算方法没有明确建模混合代码句子的 “自然性” 或 “可接受性”，但依赖于训练语料库来反映可接受的混合代码句子的分布。建模混合文本的可接受性可以帮助区分自然的混合文本，并实现质量控制的混合文本生成。为此，我们构建了 Cline 数据集，其中含有英语 - 印地语 (en-hi) 混合文本的人工可接受性判断。Cline 是其类别中最大的数据集，包含 16,642 个句子，其中包括两个来源的样本：合成混合文本和从在线社交媒体上收集的样本。我们的分析表明，用于过滤 / 筛选 / 比较混合文本语料库的流行混合代码指标（如 CMI、转换点数、突发度）与人工可接受性判断之间的相关性较低，凸显了我们数据集的必要性。使用 Cline 进行的实验证明，仅基于混合代码指标训练的简单多层感知机 (MLP) 模型被细调的预训练多语言大型语言模型 (MLLMs) 在具有挑战性的数据设置中表现更佳。具体而言，XLM-Roberta 和 Bernice 在不同配置中优于 IndicBERT。与 ChatGPT 的零和几个样本能力的比较表明，基于更大数据集进行细调的 MLLMs 优于 ChatGPT，在混合代码任务中提供了改进的空间。使用我们的模型检查点进行从英语 - 印地语到英语 - 特鲁古语的零样本转移可接受性判断，超过了随机基线，可以应用于其他混合语言对，并提供了进一步的研究方向。我们公开发布了我们的人工标注数据集、训练检查点、混合文本语料库和数据生成以及模型训练的代码。

从人类评判到预测模型：解析混合代码句子的可接受性

From Human Judgements to Predictive Models: Unravelling Acceptability in  Code-Mixed Sentences

Multilingual models have demonstrated impressive cross-lingual transfer
performance. However, test sets like XNLI are monolingual at the example level.
In multilingual communities, it is common for polyglots to code-mix when
conversing with each other. Inspired by this phenomenon, we present two strong
black-box adversarial attacks (one word-level, one phrase-level) for
multilingual models that push their ability to handle code-mixed sentences to
the limit. The former uses bilingual dictionaries to propose perturbations and
translations of the clean example for sense disambiguation. The latter directly
aligns the clean example with its translations before extracting phrases as
perturbations. Our phrase-level attack has a success rate of 89.75% against
XLM-R-large, bringing its average accuracy of 79.85 down to 8.18 on XNLI.
Finally, we propose an efficient adversarial training scheme that trains in the
same number of steps as the original model and show that it improves model
accuracy.

本文介绍两种针对多语言模型的黑盒对抗攻击和一种高效的对抗性训练方案，以提高多语言模型处理混合语言的表现。短语级攻击的成功率为 89.75％，将 XLM-R-large 的平均准确性从 79.85 降至 8.18。