Text image machine translation (TIMT) has been widely used in various
real-world applications, which translates source language texts in images into
another target language sentence. Existing methods on TIMT are mainly divided
into two categories: the recognition-then-translation pipeline model and the
end-to-end model. However, how to transfer knowledge from the pipeline model
into the end-to-end model remains an unsolved problem. In this paper, we
propose a novel Multi-Teacher Knowledge Distillation (MTKD) method to
effectively distillate knowledge into the end-to-end TIMT model from the
pipeline model. Specifically, three teachers are utilized to improve the
performance of the end-to-end TIMT model. The image encoder in the end-to-end
TIMT model is optimized with the knowledge distillation guidance from the
recognition teacher encoder, while the sequential encoder and decoder are
improved by transferring knowledge from the translation sequential and decoder
teacher models. Furthermore, both token and sentence-level knowledge
distillations are incorporated to better boost the translation performance.
Extensive experimental results show that our proposed MTKD effectively improves
the text image translation performance and outperforms existing end-to-end and
pipeline models with fewer parameters and less decoding time, illustrating that
MTKD can take advantage of both pipeline and end-to-end models.

本文提出一种基于多教师知识蒸馏 (MTKD) 的文本图像机器翻译 (TIMT) 方法，将管道模型的知识有效地转移至端到端 TIMT 模型中，包括图像编码器、序列编码器和解码器的知识引导优化，并采用令牌和句子级别知识蒸馏相结合来提高翻译性能。实验结果表明，相比于现有的管道和端到端模型，MTKD 可以更有效地改进文本图像翻译性能，具有更少的参数和更少的解码时间。

文本图像机器翻译的多教师知识蒸馏

Multi-Teacher Knowledge Distillation For Text Image Machine Translation

Text image machine translation (TIMT) aims to translate texts embedded in
images from one source language to another target language. Existing methods,
both two-stage cascade and one-stage end-to-end architectures, suffer from
different issues. The cascade models can benefit from the large-scale optical
character recognition (OCR) and MT datasets but the two-stage architecture is
redundant. The end-to-end models are efficient but suffer from training data
deficiency. To this end, in our paper, we propose an end-to-end TIMT model
fully making use of the knowledge from existing OCR and MT datasets to pursue
both an effective and efficient framework. More specifically, we build a novel
modal adapter effectively bridging the OCR encoder and MT decoder. End-to-end
TIMT loss and cross-modal contrastive loss are utilized jointly to align the
feature distribution of the OCR and MT tasks. Extensive experiments show that
the proposed method outperforms the existing two-stage cascade models and
one-stage end-to-end models with a lighter and faster architecture.
Furthermore, the ablation studies verify the generalization of our method,
where the proposed modal adapter is effective to bridge various OCR and MT
models.

本文旨在提出一种新型的端对端的文本图像翻译模型，充分利用现有的 OCR 和 MT 数据集的知识来追求既有效又高效的框架。我们建立了一种新颖的模态适配器，有效地连接 OCR 编码器和 MT 解码器，并联合使用端到端 TIMT 损失和跨模态对比损失来对齐 OCR 和 MT 任务的特征分布。广泛的实验表明，所提出的方法比现有的两阶段级联模型和一阶段端对端模型具有更轻，更快的结构，而消融研究则验证了我们的方法的泛化性。