The capability to jointly process multi-modal information is becoming an
essential task. However, the limited number of paired multi-modal data and the
large computational requirements in multi-modal learning hinder the
development. We propose a novel Tri-Modal Translation (TMT) model that
translates between arbitrary modalities spanning speech, image, and text. We
introduce a novel viewpoint, where we interpret different modalities as
different languages, and treat multi-modal translation as a well-established
machine translation problem. To this end, we tokenize speech and image data
into discrete tokens, which provide a unified interface across modalities and
significantly decrease the computational cost. In the proposed TMT, a
multi-modal encoder-decoder conducts the core translation, whereas
modality-specific processing is conducted only within the tokenization and
detokenization stages. We evaluate the proposed TMT on all six modality
translation tasks. TMT outperforms single model counterparts consistently,
demonstrating that unifying tasks is beneficial not only for practicality but
also for performance.

我们提出了一种新颖的 Tri-Modal Translation（TMT）模型，它能够在语音、图像和文本之间进行任意模态的翻译，并通过将语音和图像数据标记为离散标记来统一界面并显著降低计算成本。通过在 TMT 中使用多模态编码器 - 解码器进行核心翻译，同时仅在标记化和解标记化阶段进行模态特定的处理，我们评估了所提出的 TMT 在六个模态翻译任务上的性能，并且 TMT 始终优于单模型对应物，表明统一任务不仅在实用性上有益，而且在性能上也有益。