Recent work has shown that deeper character-based neural machine translation (NMT) models can outperform subword-based models. However, it is still unclear what makes deeper character-based models successful. In this paper, we conduct an investigation into pure character-based models in the case of translating Finnish into English, including exploring the ability to learn word senses and morphological inflections and the attention mechanism. We demonstrate that word-level information is distributed over the entire character sequence rather than over a single character, and characters at different positions play different roles in learning linguistic knowledge. In addition, character-based models need more layers to encode word senses which explains why only deeper models outperform subword-based models. The attention distribution pattern shows that separators attract a lot of attention and we explore a sparse word-level attention to enforce character hidden states to capture the full word-level information. Experimental results show that the word-level attention with a single head results in 1.2 BLEU points drop.

本文探讨了纯字符级模型在芬兰语到英语机器翻译中的效果，并证明了字符序列中不同位置的字符在学习语言知识方面扮演着不同的角色。通过实验证明，单头的基于字级别的注意力机制会导致 BLEU 分数下降 1.2 分。

纯字符级神经机器翻译的理解：以从芬兰语到英语的翻译为例