Cross-modal retrieval is the task of retrieving samples of a given modality
by using queries of a different one. Due to the wide range of practical
applications, the problem has been mainly focused on the vision and language
case, e.g. text to image retrieval, where models like CLIP have proven
effective in solving such tasks. The dominant approach to learning such
coordinated representations consists of projecting them onto a common space
where matching views stay close and those from non-matching pairs are pushed
away from each other. Although this cross-modal coordination has been applied
also to other pairwise combinations, extending it to an arbitrary number of
diverse modalities is a problem that has not been fully explored in the
literature. In this paper, we propose two different approaches to the problem.
The first is based on an extension of the CLIP contrastive objective to an
arbitrary number of input modalities, while the second departs from the
contrastive formulation and tackles the coordination problem by regressing the
cross-modal similarities towards a target that reflects two simple and
intuitive constraints of the cross-modal retrieval task. We run experiments on
two different datasets, over different combinations of input modalities and
show that the approach is not only simple and effective but also allows for
tackling the retrieval problem in novel ways. Besides capturing a more diverse
set of pair-wise interactions, we show that we can use the learned
representations to improve retrieval performance by combining the embeddings
from two or more such modalities.

提出两种不同的方法来解决跨模态检索的问题，一种基于 CLIP 对任意数量的输入模式进行扩展，而第二种方法通过回归跨模态相似性来解决协调问题，并在多个数据集上进行实验证明其简单有效，并允许以新的方式解决检索问题。