Text image machine translation (TIMT) aims to translate texts embedded in
images from one source language to another target language. Existing methods,
both two-stage cascade and one-stage end-to-end architectures, suffer from
different issues. The cascade models can benefit from the large-scale optical
character recognition (OCR) and MT datasets but the two-stage architecture is
redundant. The end-to-end models are efficient but suffer from training data
deficiency. To this end, in our paper, we propose an end-to-end TIMT model
fully making use of the knowledge from existing OCR and MT datasets to pursue
both an effective and efficient framework. More specifically, we build a novel
modal adapter effectively bridging the OCR encoder and MT decoder. End-to-end
TIMT loss and cross-modal contrastive loss are utilized jointly to align the
feature distribution of the OCR and MT tasks. Extensive experiments show that
the proposed method outperforms the existing two-stage cascade models and
one-stage end-to-end models with a lighter and faster architecture.
Furthermore, the ablation studies verify the generalization of our method,
where the proposed modal adapter is effective to bridge various OCR and MT
models.

本文旨在提出一种新型的端对端的文本图像翻译模型，充分利用现有的 OCR 和 MT 数据集的知识来追求既有效又高效的框架。我们建立了一种新颖的模态适配器，有效地连接 OCR 编码器和 MT 解码器，并联合使用端到端 TIMT 损失和跨模态对比损失来对齐 OCR 和 MT 任务的特征分布。广泛的实验表明，所提出的方法比现有的两阶段级联模型和一阶段端对端模型具有更轻，更快的结构，而消融研究则验证了我们的方法的泛化性。