Byte pair encoding (BPE) emerges as an effective tokenization method for
tackling the out-of-vocabulary (OOV) challenge in various natural language and
speech processing tasks. Recent research highlights the dependency of BPE
subword tokenization's efficacy on the morphological nature of the language,
particularly in languages rich in inflectional morphology, where fewer BPE
merges suffice for generating highly productive tokens. Motivated by this, our
study empirically identifies the optimal number of BPE tokens for Bengali, a
language known for its morphological complexity, thus enhancing
out-of-distribution automatic speech recognition (ASR) performance.
Experimental evaluation reveals that an excessively high number of BPE tokens
can lead to overfitting, while approximately 500-1000 tokens result in superior
OOV performance. Furthermore, we conduct a comparative analysis of BPE with
character-based and unigram-based tokenization methods. By introducing BPE
tokenization to Bengali ASR, we achieve a substantial reduction in the word
error rate (WER) from 66.44% in our character-based baseline system to 63.80%
on the LB-ASRTD eval set and from 46.34% to 42.80% on the SHRUTI eval set, both
of which include out-of-distribution data.

通过实验证明，对于孟加拉语，Byte pair encoding (BPE) 是一种在自动语音识别（ASR）中改善识别性能的有效分词方法，较佳的 BPE token 数量为大约 500-1000 个。

字节对编码是自动孟加拉语语音识别所需的全部

Byte Pair Encoding Is All You Need For Automatic Bengali Speech  Recognition

Having recognized gender bias as a major issue affecting current translation
technologies, researchers have primarily attempted to mitigate it by working on
the data front. However, whether algorithmic aspects concur to exacerbate
unwanted outputs remains so far under-investigated. In this work, we bring the
analysis on gender bias in automatic translation onto a seemingly neutral yet
critical component: word segmentation. Can segmenting methods influence the
ability to translate gender? Do certain segmentation approaches penalize the
representation of feminine linguistic markings? We address these questions by
comparing 5 existing segmentation strategies on the target side of speech
translation systems. Our results on two language pairs (English-Italian/French)
show that state-of-the-art sub-word splitting (BPE) comes at the cost of higher
gender bias. In light of this finding, we propose a combined approach that
preserves BPE overall translation quality, while leveraging the higher ability
of character-based segmentation to properly translate gender.

本文探讨了中性的分词组件是否会影响到翻译性别，以及某些分词方法是否会惩罚女性语言标记的表示。作者比较了 5 种不同的分词策略，并发现最先进的子单词拆分技术 (BPE) 导致更高的性别偏见。因此，我们提出一种结合字符级分词更好地翻译性别，同时保留 BPE 整体翻译质量的方法。

分词对语音翻译性别偏差的影响

How to Split: the Effect of Word Segmentation on Gender Bias in Speech  Translation

This paper evaluates the performance of several modern subword segmentation
methods in a low-resource neural machine translation setting. We compare
segmentations produced by applying BPE at the token or sentence level with
morphologically-based segmentations from LMVR and MORSEL. We evaluate
translation tasks between English and each of Nepali, Sinhala, and Kazakh, and
predict that using morphologically-based segmentation methods would lead to
better performance in this setting. However, comparing to BPE, we find that no
consistent and reliable differences emerge between the segmentation methods.
While morphologically-based methods outperform BPE in a few cases, what
performs best tends to vary across tasks, and the performance of segmentation
methods is often statistically indistinguishable.

比较了基于形态学和 BPE 方法的子词切分对低资源神经机器翻译任务的影响，发现其对翻译性能影响不一，并且各自有表现最好的情况。