Contrastive Language-Image Pretraining (CLIP) is widely used to train models
to align images and texts in a common embedding space by mapping them to
fixed-sized vectors. These models are key to multimodal information retrieval
and related tasks. However, CLIP models generally underperform in text-only
tasks compared to specialized text models. This creates inefficiencies for
information retrieval systems that keep separate embeddings and models for
text-only and multimodal tasks. We propose a novel, multi-task contrastive
training method to address this issue, which we use to train the jina-clip-v1
model to achieve the state-of-the-art performance on both text-image and
text-text retrieval tasks.

通过多任务对比训练方法，我们提出了一种新颖的 jina-clip-v1 模型训练方法，以在文本 - 图像和文本 - 文本检索任务上实现了最先进的性能。

Jina CLIP：你的 CLIP 模型也是你的文本检索器

Jina CLIP: Your CLIP Model Is Also Your Text Retriever

Two modalities are often used to convey information in a complementary and
beneficial manner, e.g., in online news, videos, educational resources, or
scientific publications. The automatic understanding of semantic correlations
between text and associated images as well as their interplay has a great
potential for enhanced multimodal web search and recommender systems. However,
automatic understanding of multimodal information is still an unsolved research
problem. Recent approaches such as image captioning focus on precisely
describing visual content and translating it to text, but typically address
neither semantic interpretations nor the specific role or purpose of an
image-text constellation. In this paper, we go beyond previous work and
investigate, inspired by research in visual communication, useful semantic
image-text relations for multimodal information retrieval. We derive a
categorization of eight semantic image-text classes (e.g., "illustration" or
"anchorage") and show how they can systematically be characterized by a set of
three metrics: cross-modal mutual information, semantic correlation, and the
status relation of image and text. Furthermore, we present a deep learning
system to predict these classes by utilizing multimodal embeddings. To obtain a
sufficiently large amount of training data, we have automatically collected and
augmented data from a variety of data sets and web resources, which enables
future research on this topic. Experimental results on a demanding test set
demonstrate the feasibility of the approach.

本文借鉴视觉传达研究，探究了多模式信息检索的有用语义图像 - 文本关系，在自动收集和扩充数据资源的基础上，采用深度学习系统和三种度量标准（跨模态互信息、语义相关性和图像与文本的状态关系）对八种语义图像 - 文本类别进行了预测，并在一个严格的测试集上展示了该方法的可行性。