Contrastive Language--Image Pre-training (CLIP) has manifested remarkable
improvements in zero-shot classification and cross-modal vision-language tasks.
Yet, from a geometrical point of view, the CLIP embedding space has been found
to have a pronounced modality gap. This gap renders the embedding space overly
sparse and disconnected, with different modalities being densely distributed in
distinct subregions of the hypersphere. In this work, we aim at answering two
main questions: 1. Does sharing the parameter space between the multi-modal
encoders reduce the modality gap? 2. Can the gap be mitigated by pushing apart
the uni-modal embeddings via intra-modality separation? We design AlignCLIP, in
order to answer these questions and show that answers to both questions are
positive. Through extensive experiments, we show that AlignCLIP achieves
noticeable enhancements in the cross-modal alignment of the embeddings, and
thereby, reduces the modality gap, while maintaining the performance across
several downstream evaluations, such as zero-shot image classification,
zero-shot multi-modal retrieval and zero-shot semantic text similarity.

通过设计 AlignCLIP，本文回答了共享多模态编码器参数空间是否减少模态间隙以及通过内部模态分离是否可以缓解间隙的两个主要问题，并通过大量实验证明了 AlignCLIP 在嵌入的跨模态对齐上取得了显著的改进，从而减少了模态间隙，并在零样本图像分类、零样本多模态检索和零样本语义文本相似性等多个下游评估中保持了性能。

缓解差距：研究提升 CLIP 中跨模态对齐的方法

Mitigate the Gap: Investigating Approaches for Improving Cross-Modal  Alignment in CLIP

Recently, the astonishing performance of large language models (LLMs) in
natural language comprehension and generation tasks triggered lots of
exploration of using them as central controllers to build agent systems.
Multiple studies focus on bridging the LLMs to external tools to extend the
application scenarios. However, the current LLMs' perceiving tool-use ability
is limited to a single text query, which may result in ambiguity in
understanding the users' real intentions. LLMs are expected to eliminate that
by perceiving the visual- or auditory-grounded instructions' information.
Therefore, in this paper, we propose Tool-LMM, a system incorporating
open-source LLMs and multi-modal encoders so that the learnt LLMs can be
conscious of multi-modal input instruction and then select the function-matched
tool correctly. To facilitate the evaluation of the model's capability, we
collect a dataset featured by consisting of multi-modal input tools from
HuggingFace. Another important feature of our dataset is that our dataset also
contains multiple potential choices for the same instruction due to the
existence of identical functions and synonymous functions, which provides more
potential solutions for the same query. The experiments reveal that our LMM is
capable of recommending appropriate tools for multi-modal instructions. Codes
and data are available at this https URL

通过使用多模态编码器将开源大语言模型（LLM）与多模态输入指令结合起来，我们提出了 Tool-LMM 系统，使学习的 LLMs 能够意识到多模态输入指令并正确选择匹配功能的工具，实验证明我们的 LMM 能够为多模态指令推荐适当的工具。

Tool-LMM：一个用于工具智能学习的大型多模态模型

Tool-LMM: A Large Multi-Modal Model for Tool Agent Learning

Pretrained unimodal encoders incorporate rich semantic information into
embedding space structures. To be similarly informative, multi-modal encoders
typically require massive amounts of paired data for alignment and training. We
introduce a semi-supervised Geometrically Regularized Alignment (GeRA) method
to align the embedding spaces of pretrained unimodal encoders in a
label-efficient way. Our method leverages the manifold geometry of unpaired
(unlabeled) data to improve alignment performance. To prevent distortions to
local geometry during the alignment process, potentially disrupting semantic
neighborhood structures and causing misalignment of unobserved pairs, we
introduce a geometric loss term. This term is built upon a diffusion operator
that captures the local manifold geometry of the unimodal pretrained encoders.
GeRA is modality-agnostic and thus can be used to align pretrained encoders
from any data modalities. We provide empirical evidence to the effectiveness of
our method in the domains of speech-text and image-text alignment. Our
experiments demonstrate significant improvement in alignment quality compared
to a variaty of leading baselines, especially with a small amount of paired
data, using our proposed geometric regularization.

引入半监督几何规范化对齐方法（GeRA）来对齐预训练单模态编码器的嵌入空间，通过利用不配对（无标签）数据的流形几何来改善对齐性能，引入了基于扩散算子的几何损失项来防止对局部几何的扭曲，GeRA 是模态不可知的，可以用于对来自任何数据模态的预训练编码器进行对齐，我们的实验证明，在语音 - 文本和图像 - 文本对齐领域，我们的方法较一些主流基线方法在对齐质量方面取得了显著改善，特别是在有限的配对数据量下，使用我们提出的几何规范化方法。

GeRA：标签高效的几何正则化对齐

GeRA: Label-Efficient Geometrically Regularized Alignment

Multi-modal encoders map images, sounds, texts, videos, etc. into a single
embedding space, aligning representations across modalities (e.g., associate an
image of a dog with a barking sound). We show that multi-modal embeddings can
be vulnerable to an attack we call "adversarial illusions." Given an input in
any modality, an adversary can perturb it so as to make its embedding close to
that of an arbitrary, adversary-chosen input in another modality. Illusions
thus enable the adversary to align any image with any text, any text with any
sound, etc.
Adversarial illusions exploit proximity in the embedding space and are thus
agnostic to downstream tasks. Using ImageBind embeddings, we demonstrate how
adversarially aligned inputs, generated without knowledge of specific
downstream tasks, mislead image generation, text generation, and zero-shot
classification.

多模态编码器、敌对幻觉、嵌入空间、接近度、零样本分类。