Machine Translation (MT) is one of the most prominent tasks in Natural
Language Processing (NLP) which involves the automatic conversion of texts from
one natural language to another while preserving its meaning and fluency.
Although the research in machine translation has been going on since multiple
decades, the newer approach of integrating deep learning techniques in natural
language processing has led to significant improvements in the translation
quality. In this paper, we have developed a Neural Machine Translation (NMT)
system by training the Transformer model to translate texts from Indian
Language Hindi to English. Hindi being a low resource language has made it
difficult for neural networks to understand the language thereby leading to a
slow growth in the development of neural machine translators. Thus, to address
this gap, we implemented back-translation to augment the training data and for
creating the vocabulary, we experimented with both word and subword level
tokenization using Byte Pair Encoding (BPE) thereby ending up training the
Transformer in 10 different configurations. This led us to achieve a
state-of-the-art BLEU score of 24.53 on the test set of IIT Bombay
English-Hindi Corpus in one of the configurations.

我们使用深度学习技术开发了一种神经机器翻译系统，通过训练 Transformer 模型将印度语 Hindi 翻译成英语，通过实施回译增强训练数据并尝试使用 Byte Pair Encoding (BPE) 进行词汇划分，我们在 10 个不同的配置中训练 Transformer，在 IIT Bombay 英印语语料库的测试集中取得了 24.53 的最新 BLEU 分数。

基于 Transformer 的印地语到英语神经机器翻译

Hindi to English: Transformer-Based Neural Machine Translation

Typically, tokenization is the very first step in most text processing works.
As a token serves as an atomic unit that embeds the contextual information of
text, how to define a token plays a decisive role in the performance of a
model.Even though Byte Pair Encoding (BPE) has been considered the de facto
standard tokenization method due to its simplicity and universality, it still
remains unclear whether BPE works best across all languages and tasks. In this
paper, we test several tokenization strategies in order to answer our primary
research question, that is, "What is the best tokenization strategy for Korean
NLP tasks?" Experimental results demonstrate that a hybrid approach of
morphological segmentation followed by BPE works best in Korean to/from English
machine translation and natural language understanding tasks such as KorNLI,
KorSTS, NSMC, and PAWS-X. As an exception, for KorQuAD, the Korean extension of
SQuAD, BPE segmentation turns out to be the most effective.

测试出，对于韩国自然语言处理任务，采用基于词形的分词再使用 BPE 的混合方法效果最佳。