Being a popular mode of text-based communication in multilingual communities,
code-mixing in online social media has became an important subject to study.
Learning the semantics and morphology of code-mixed language remains a key
challenge, due to scarcity of data and unavailability of robust and
language-invariant representation learning technique. Any morphologically-rich
language can benefit from character, subword, and word-level embeddings, aiding
in learning meaningful correlations. In this paper, we explore a hierarchical
transformer-based architecture (HIT) to learn the semantics of code-mixed
languages. HIT consists of multi-headed self-attention and outer product
attention components to simultaneously comprehend the semantic and syntactic
structures of code-mixed texts. We evaluate the proposed method across 6 Indian
languages (Bengali, Gujarati, Hindi, Tamil, Telugu and Malayalam) and Spanish
for 9 NLP tasks on 17 datasets. The HIT model outperforms state-of-the-art
code-mixed representation learning and multilingual language models in all
tasks. We further demonstrate the generalizability of the HIT architecture
using masked language modeling-based pre-training, zero-shot learning, and
transfer learning approaches. Our empirical results show that the pre-training
objectives significantly improve the performance on downstream tasks.

采用分层变压器的架构（HIT）学习混合代码语言的语义和语法结构，在 17 个数据集和 9 个自然语言处理任务中，HIT 模型都表现出超越最先进的代码混合表示学习和多语言模型的性能。

利用分层 Transformer 深入理解混合编码语言语义

A Comprehensive Understanding of Code-mixed Language Semantics using Hierarchical Transformer

Code-switching, the alternation of languages within a conversation or
utterance, is a common communicative phenomenon that occurs in multilingual
communities across the world. This survey reviews computational approaches for
code-switched Speech and Natural Language Processing. We motivate why
processing code-switched text and speech is essential for building intelligent
agents and systems that interact with users in multilingual communities. As
code-switching data and resources are scarce, we list what is available in
various code-switched language pairs with the language processing tasks they
can be used for. We review code-switching research in various Speech and NLP
applications, including language processing tools and end-to-end systems. We
conclude with future directions and open problems in the field.

本篇论文综述了多语言社区中通用的一种交际现象 —— 语码转换，并且概述了计算机语言处理的方法。同时该文阐述了处理语码转换的语音和自然语言处理在建立智能代理和与多语言社区用户的交互系统中的重要作用，发现语码转换数据和资源的稀缺性，并列出了可利用的各种编码语言对及相应的自然语言处理任务。最后，概述了语码转换在各种语音和自然语言处理应用中的研究，并对该领域的未来方向和问题进行了总结。