Though dialectal language is increasingly abundant on social media, few
resources exist for developing NLP tools to handle such language. We conduct a
case study of dialectal language in online conversational text by investigating
African-American English (AAE) on Twitter. We propose a distantly supervised
model to identify AAE-like language from demographics associated with
geo-located messages, and we verify that this language follows well-known AAE
linguistic phenomena. In addition, we analyze the quality of existing language
identification and dependency parsing tools on AAE-like text, demonstrating
that they perform poorly on such text compared to text associated with white
speakers. We also provide an ensemble classifier for language identification
which eliminates this disparity and release a new corpus of tweets containing
AAE-like language.

本文研究表明，虽然方言语言在社交媒体上越来越普遍，但是开发自然语言处理工具的资源很少。本文以推特上的非裔美国英语为案例研究方言语言的实用性。文章提出了一种基于地理位置信息的远距离语言学习模型来识别 AAEL（African-American English like language），并验证了这种语言遵循已知的非裔美国英语语言现象。此外，文章对现有的语言识别和依赖解析工具在 AAEL 文本上的质量进行了分析，证明它们在处理这样的文本时比白人说话者的文本表现差。作者还提供了一种语言识别的整合分类器，消除了这种差异，并发布了一个包含 AAEL 类似语言的新推文语料库。

社交媒体中的人口方言差异：非裔美国英语案例研究

Demographic Dialectal Variation in Social Media: A Case Study of  African-American English

Social media messages' brevity and unconventional spelling pose a challenge
to language identification. We introduce a hierarchical model that learns
character and contextualized word-level representations for language
identification. Our method performs well against strong base- lines, and can
also reveal code-switching.

本文介绍了一种基于层次模型的语言识别方法，其中字符级和上下文化的词级表示能够很好地处理社交媒体消息的简洁特性和非传统拼写，同时还能揭示代码开关现象。

层次字符 - 词语模型在语言识别中的应用

Hierarchical Character-Word Models for Language Identification

Random Indexing is a simple implementation of Random Projections with a wide
range of applications. It can solve a variety of problems with good accuracy
without introducing much complexity. Here we use it for identifying the
language of text samples. We present a novel method of generating language
representation vectors using letter blocks. Further, we show that the method is
easily implemented and requires little computational power and space.
Experiments on a number of model parameters illustrate certain properties about
high dimensional sparse vector representations of data. Proof of statistically
relevant language vectors are shown through the extremely high success of
various language recognition tasks. On a difficult data set of 21,000 short
sentences from 21 different languages, our model performs a language
recognition task and achieves 97.8% accuracy, comparable to state-of-the-art
methods.

使用随机投影和随机向量生成较为简单的随机索引，分析其在语言识别中的应用，证明其在 21 门不同语言识别中取得了 97.8% 的高精度。

利用随机索引进行语言识别

Language Recognition using Random Indexing

The movements of ideas and content between locations and languages are
unquestionably crucial concerns to researchers of the information age, and
Twitter has emerged as a central, global platform on which hundreds of millions
of people share knowledge and information. A variety of research has attempted
to harvest locational and linguistic metadata from tweets in order to
understand important questions related to the 300 million tweets that flow
through the platform each day. However, much of this work is carried out with
only limited understandings of how best to work with the spatial and linguistic
contexts in which the information was produced. Furthermore, standard,
well-accepted practices have yet to emerge. As such, this paper studies the
reliability of key methods used to determine language and location of content
in Twitter. It compares three automated language identification packages to
Twitter's user interface language setting and to a human coding of languages in
order to identify common sources of disagreement. The paper also demonstrates
that in many cases user-entered profile locations differ from the physical
locations users are actually tweeting from. As such, these open-ended,
user-generated, profile locations cannot be used as useful proxies for the
physical locations from which information is published to Twitter.

本文研究了三种自动语言识别方法与 Twitter 用户界面语言设置和语言人工编码之间的可靠性，比较了用户输入的个人资料位置与实际发推地点之间的不同，证明了用户生成的资料位置无法用作推特信息发布的有用代理。