Word translation without parallel corpora has become feasible, rivaling the
performance of supervised methods. Recent findings have shown that the accuracy
and robustness of unsupervised word translation (UWT) can be improved by making
use of visual observations, which are universal representations across
languages. In this work, we investigate the potential of using not only visual
observations but also pretrained language-image models for enabling a more
efficient and robust UWT. Specifically, we develop a novel UWT method dubbed
Word Alignment using Language-Image Pretraining (WALIP), which leverages visual
observations via the shared embedding space of images and texts provided by
CLIP models (Radford et al., 2021). WALIP has a two-step procedure. First, we
retrieve word pairs with high confidences of similarity, computed using our
proposed image-based fingerprints, which define the initial pivot for the word
alignment. Second, we apply our robust Procrustes algorithm to estimate the
linear mapping between two embedding spaces, which iteratively corrects and
refines the estimated alignment. Our extensive experiments show that WALIP
improves upon the state-of-the-art performance of bilingual word alignment for
a few language pairs across different word embeddings and displays great
robustness to the dissimilarity of language pairs or training corpora for two
word embeddings.

本文提出了一种名为 WALIP 的 UWT 方法，它基于预训练的语言 - 图像模型和 CLIP 模型，使用图像指纹来定义初始节点，再通过鲁棒的 Procrustes 算法迭代校正和细化估计的对齐，此方法在语言对和词向量不同的情况下表现出了极大的鲁棒性。