In-image machine translation (IIMT) aims to translate an image containing texts in source language into an image containing translations in target language. In this regard, conventional cascaded methods suffer from issues such as error propagation, massive parameters, and difficulties in deployment and retaining visual characteristics of the input image. Thus, constructing end-to-end models has become an option, which, however, faces two main challenges: 1) the huge modeling burden, as it is required to simultaneously learn alignment across languages and preserve the visual characteristics of the input image; 2) the difficulties of directly predicting excessively lengthy pixel sequences. In this paper, we propose \textit{Translatotron-V(ision)}, an end-to-end IIMT model consisting of four modules. In addition to an image encoder, and an image decoder, our model contains a target text decoder and an image tokenizer. Among them, the target text decoder is used to alleviate the language alignment burden, and the image tokenizer converts long sequences of pixels into shorter sequences of visual tokens, preventing the model from focusing on low-level visual features. Besides, we present a two-stage training framework for our model to assist the model in learning alignment across modalities and languages. Finally, we propose a location-aware evaluation metric called Structure-BLEU to assess the translation quality of the generated images. Experimental results demonstrate that our model achieves competitive performance compared to cascaded models with only 70.9\% of parameters, and significantly outperforms the pixel-level end-to-end IIMT model.

Translatotron-V(ision)是一个端到端的图像机器翻译模型，通过四个模块实现，其中包括图像编码器、图像解码器、目标文本解码器和图像分词器，并采用两阶段训练框架以提高模型对多模态和多语言的对齐能力，同时引入结构化BLEU作为评估生成图像翻译质量的指标。实验证明，该模型在与级联模型参数仅占70.9%的情况下取得了有竞争力的性能，并且显著优于像素级端到端的图像机器翻译模型。

Translatotron-V(ison): 图像内机器翻译的端到端模型