Language identification is used as the first step in many data collection and
crawling efforts because it allows us to sort online text into
language-specific buckets. However, many modern languages, such as Konkani,
Kashmiri, Punjabi etc., are synchronically written in several scripts.
Moreover, languages with different writing systems do not share significant
lexical, semantic, and syntactic properties in neural representation spaces,
which is a disadvantage for closely related languages and low-resource
languages, especially those from the Indian Subcontinent. To counter this, we
propose learning script-agnostic representations using several different
experimental strategies (upscaling, flattening, and script mixing) focusing on
four major Dravidian languages (Tamil, Telugu, Kannada, and Malayalam). We find
that word-level script randomization and exposure to a language written in
multiple scripts is extremely valuable for downstream script-agnostic language
identification, while also maintaining competitive performance on naturally
occurring text.

我们提出使用多种实验策略（放大、扁平化和混合脚本）来学习与脚本无关的表示，重点关注四种主要的德拉维达语言（泰米尔语、泰卢固语、卡纳达语和马拉雅拉姆语），发现在下游的不以脚本为依据的语言识别中，词级脚本随机化和暴露于多个脚本书写的语言对于维持竞争性自然文本性能也非常有价值。