In this study, a human evaluation is carried out on how hyperparameter
settings impact the quality of Transformer-based Neural Machine Translation
(NMT) for the low-resourced English--Irish pair. SentencePiece models using
both Byte Pair Encoding (BPE) and unigram approaches were appraised. Variations
in model architectures included modifying the number of layers, evaluating the
optimal number of heads for attention and testing various regularisation
techniques. The greatest performance improvement was recorded for a
Transformer-optimized model with a 16k BPE subword model. Compared with a
baseline Recurrent Neural Network (RNN) model, a Transformer-optimized model
demonstrated a BLEU score improvement of 7.8 points. When benchmarked against
Google Translate, our translation engines demonstrated significant
improvements. Furthermore, a quantitative fine-grained manual evaluation was
conducted which compared the performance of machine translation systems. Using
the Multidimensional Quality Metrics (MQM) error taxonomy, a human evaluation
of the error types generated by an RNN-based system and a Transformer-based
system was explored. Our findings show the best-performing Transformer system
significantly reduces both accuracy and fluency errors when compared with an
RNN-based model.

该研究评估了超参数设置对基于 Transformer 的神经机器翻译在英语 - 爱尔兰语低资源对中的质量的影响。研究中使用了基于 Byte Pair Encoding (BPE) 和 unigram 方法的 SentencePiece 模型。通过修改层数、评估注意力机制中最佳头数、使用不同的正则化技术等变体，发现使用 16k BPE 子词模型优化的 Transformer 模型表现最佳，相较于基线递归神经网络（RNN）模型，BLEU 分数提高了 7.8 个点，与 Google 翻译相比，该翻译引擎实现了显著提升。此外，还通过定量细粒度手动评估，比较了机器翻译系统的表现。使用多维质量度量中的错误分类法（MQM error taxonomy），探讨了基于 RNN 模型和 Transformer 模型生成的错误类型的人类评估。结果表明，表现最佳的 Transformer 系统在准确性和流畅性错误方面与基于 RNN 的模型相比均有显著降低。