The study investigates the effectiveness of utilizing multimodal information
in Neural Machine Translation (NMT). While prior research focused on using
multimodal data in low-resource scenarios, this study examines how image
features impact translation when added to a large-scale, pre-trained unimodal
NMT system. Surprisingly, the study finds that images might be redundant in
this context. Additionally, the research introduces synthetic noise to assess
whether images help the model deal with textual noise. Multimodal models
slightly outperform text-only models in noisy settings, even with random
images. The study's experiments translate from English to Hindi, Bengali, and
Malayalam, outperforming state-of-the-art benchmarks significantly.
Interestingly, the effect of visual context varies with source text noise: no
visual context works best for non-noisy translations, cropped image features
are optimal for low noise, and full image features work better in high-noise
scenarios. This sheds light on the role of visual context, especially in noisy
settings, opening up a new research direction for Noisy Neural Machine
Translation in multimodal setups. The research emphasizes the importance of
combining visual and textual information for improved translation in various
environments.

利用多模态信息在神经机器翻译 (NMT) 中的有效性进行了研究，发现图像在大规模预训练的单模态 NMT 系统中可能是多余的，同时介绍了合成噪声来评估图像是否有助于模型处理文本噪声，多模态模型在嘈杂环境中微弱地胜过纯文本模型，实验证明视觉背景的效果与源文本噪声有关，对噪声神经机器翻译在多模态设置中的作用有所启示，强调了在各种环境中结合视觉和文本信息以改进翻译质量的重要性。

视觉背景对噪声多模态神经机器翻译的影响：英语到印度语言的实证研究

Impact of Visual Context on Noisy Multimodal NMT: An Empirical Study for  English to Indian Languages

Textual noise, such as typos or abbreviations, is a well-known issue that
penalizes vanilla Transformers for most downstream tasks. We show that this is
also the case for sentence similarity, a fundamental task in multiple domains,
e.g. matching, retrieval or paraphrasing. Sentence similarity can be approached
using cross-encoders, where the two sentences are concatenated in the input
allowing the model to exploit the inter-relations between them. Previous works
addressing the noise issue mainly rely on data augmentation strategies, showing
improved robustness when dealing with corrupted samples that are similar to the
ones used for training. However, all these methods still suffer from the token
distribution shift induced by typos. In this work, we propose to tackle textual
noise by equipping cross-encoders with a novel LExical-aware Attention module
(LEA) that incorporates lexical similarities between words in both sentences.
By using raw text similarities, our approach avoids the tokenization shift
problem obtaining improved robustness. We demonstrate that the attention bias
introduced by LEA helps cross-encoders to tackle complex scenarios with textual
noise, specially in domains with short-text descriptions and limited context.
Experiments using three popular Transformer encoders in five e-commerce
datasets for product matching show that LEA consistently boosts performance
under the presence of noise, while remaining competitive on the original
(clean) splits. We also evaluate our approach in two datasets for textual
entailment and paraphrasing showing that LEA is robust to typos in domains with
longer sentences and more natural context. Additionally, we thoroughly analyze
several design choices in our approach, providing insights about the impact of
the decisions made and fostering future research in cross-encoders dealing with
typos.

通过引入词汇感知注意力模块（LEA）来解决文本噪音问题，增强交叉编码器在具有短文本描述和有限上下文的复杂场景中应对文本噪音；在商品匹配和文本蕴含等领域中，LEA 通过考虑词汇相似性对打字错误鲁棒性进行改进，在干净数据集和包含噪音数据集上均展现出竞争力，并为交叉编码器处理打字错误的研究提供了启示。