Accurate transcription of Bengali text to the International Phonetic Alphabet
(IPA) is a challenging task due to the complex phonology of the language and
context-dependent sound changes. This challenge is even more for regional
Bengali dialects due to unavailability of standardized spelling conventions for
these dialects, presence of local and foreign words popular in those regions
and phonological diversity across different regions. This paper presents an
approach to this sequence-to-sequence problem by introducing the District
Guided Tokens (DGT) technique on a new dataset spanning six districts of
Bangladesh. The key idea is to provide the model with explicit information
about the regional dialect or "district" of the input text before generating
the IPA transcription. This is achieved by prepending a district token to the
input sequence, effectively guiding the model to understand the unique phonetic
patterns associated with each district. The DGT technique is applied to
fine-tune several transformer-based models, on this new dataset. Experimental
results demonstrate the effectiveness of DGT, with the ByT5 model achieving
superior performance over word-based models like mT5, BanglaT5, and umT5. This
is attributed to ByT5's ability to handle a high percentage of
out-of-vocabulary words in the test set. The proposed approach highlights the
importance of incorporating regional dialect information into ubiquitous
natural language processing systems for languages with diverse phonological
variations. The following work was a result of the "Bhashamul" challenge, which
is dedicated to solving the problem of Bengali text with regional dialects to
IPA transcription this https URL The training and
inference notebooks are available through the competition link.

该论文介绍了一种将孟加拉语文本准确转录为国际音标的方法，通过在输入序列之前加入地区标记，模型能够理解每个地区的独特语音模式，实验证明该方法在处理多样化语音变化的语言中具有显著效果。

使用区域性方言将孟加拉文字转写为国际音标的方法

Transcribing Bengali Text with Regional Dialects to IPA using District  Guided Tokens

The Bangla linguistic variety is a fascinating mix of regional dialects that
adds to the cultural diversity of the Bangla-speaking community. Despite
extensive study into translating Bangla to English, English to Bangla, and
Banglish to Bangla in the past, there has been a noticeable gap in translating
Bangla regional dialects into standard Bangla. In this study, we set out to
fill this gap by creating a collection of 32,500 sentences, encompassing
Bangla, Banglish, and English, representing five regional Bangla dialects. Our
aim is to translate these regional dialects into standard Bangla and detect
regions accurately. To achieve this, we proposed models known as mT5 and
BanglaT5 for translating regional dialects into standard Bangla. Additionally,
we employed mBERT and Bangla-bert-base to determine the specific regions from
where these dialects originated. Our experimental results showed the highest
BLEU score of 69.06 for Mymensingh regional dialects and the lowest BLEU score
of 36.75 for Chittagong regional dialects. We also observed the lowest average
word error rate of 0.1548 for Mymensingh regional dialects and the highest of
0.3385 for Chittagong regional dialects. For region detection, we achieved an
accuracy of 85.86% for Bangla-bert-base and 84.36% for mBERT. This is the first
large-scale investigation of Bangla regional dialects to Bangla machine
translation. We believe our findings will not only pave the way for future work
on Bangla regional dialects to Bangla machine translation, but will also be
useful in solving similar language-related challenges in low-resource language
conditions.

本研究通过创建一个包含 32,500 个句子的语料库，涵盖孟加拉地区的 5 种地方性孟加拉方言，提出了一种将这些方言翻译为标准孟加拉语并准确检测来源地区的模型 mT5 和 BanglaT5。实验结果显示，孟加拉地区方言的 BLEU 得分最高为 69.06，Chittagong 地区方言的 BLEU 得分最低为 36.75，Mymensingh 地区方言的平均词错误率最低为 0.1548，Chittagong 地区方言的平均词错误率最高为 0.3385。对于地区检测，使用 Bangla-bert-base 和 mBERT 分别达到了 85.86% 和 84.36% 的准确率。该研究为孟加拉方言到孟加拉机器翻译提供了首次大规模的调查，并为解决类似资源匮乏语言环境中的语言相关挑战提供了重要参考。

Vashantor：用于孟加拉区域方言自动翻译到孟加拉语的大规模多语种基准数据集

Vashantor: A Large-scale Multilingual Benchmark Dataset for Automated  Translation of Bangla Regional Dialects to Bangla Language

Historically, researchers and consumers have noticed a decrease in quality
when applying NLP tools to minority variants of languages (i.e. Puerto Rican
Spanish or Swiss German), but studies exploring this have been limited to a
select few languages. Additionally, past studies have mainly been conducted in
a monolingual context, so cross-linguistic trends have not been identified and
tied to external factors. In this work, we conduct a comprehensive evaluation
of the most influential, state-of-the-art large language models (LLMs) across
two high-use applications, machine translation and automatic speech
recognition, to assess their functionality on the regional dialects of several
high- and low-resource languages. Additionally, we analyze how the regional
dialect gap is correlated with economic, social, and linguistic factors. The
impact of training data, including related factors like dataset size and its
construction procedure, is shown to be significant but not consistent across
models or languages, meaning a one-size-fits-all approach cannot be taken in
solving the dialect gap. This work will lay the foundation for furthering the
field of dialectal NLP by laying out evident disparities and identifying
possible pathways for addressing them through mindful data collection.

本文通过全面评估最有影响力的最新大型语言模型（LLMs）在机器翻译和自动语音识别两个高使用频率应用领域上的功能，对多个高和低资源语言的地区方言进行了功能评估，并分析了地方方言差距与经济、社会和语言因素的相关性，从而为方言 NLP 领域的发展奠定基础，并通过有意识的数据收集，揭示明显的差异并寻找可能的解决途径。