Code-mixing is the practice of using two or more languages in a single sentence, which often occurs in multilingual communities such as India where people commonly speak multiple languages. Classic NLP tools, trained on monolingual data, face challenges when dealing with code-mixed data. Extracting meaningful information from sentences containing multiple languages becomes difficult, particularly in tasks like hate speech detection, due to linguistic variation, cultural nuances, and data sparsity. To address this, we aim to analyze the significance of code-mixed embeddings and evaluate the performance of BERT and HingBERT models (trained on a Hindi-English corpus) in hate speech detection. Our study demonstrates that HingBERT models, benefiting from training on the extensive Hindi-English dataset L3Cube-HingCorpus, outperform BERT models when tested on hate speech text datasets. We also found that code-mixed Hing-FastText performs better than standard English FastText and vanilla BERT models.

本研究解决了传统自然语言处理工具在处理混合语言数据时面临的挑战，特别是在仇恨言论检测任务中的问题。我们通过分析混合语言嵌入的重要性，发现HingBERT模型在Hindi-English数据集的训练后，在仇恨言论文本数据集上的表现优于BERT模型。此项研究表明，混合语言模型在多语言社区中的应用潜力巨大。

混合语言嵌入在仇恨言论识别中的重要性