As Artificial General Intelligence (AGI) becomes increasingly integrated into
various facets of human life, ensuring the safety and ethical alignment of such
systems is paramount. Previous studies primarily focus on single-modality
threats, which may not suffice given the integrated and complex nature of
cross-modality interactions. We introduce a novel safety alignment challenge
called Safe Inputs but Unsafe Output (SIUO) to evaluate cross-modality safety
alignment. Specifically, it considers cases where single modalities are safe
independently but could potentially lead to unsafe or unethical outputs when
combined. To empirically investigate this problem, we developed the SIUO, a
cross-modality benchmark encompassing 9 critical safety domains, such as
self-harm, illegal activities, and privacy violations. Our findings reveal
substantial safety vulnerabilities in both closed- and open-source LVLMs, such
as GPT-4V and LLaVA, underscoring the inadequacy of current models to reliably
interpret and respond to complex, real-world scenarios.

人工通用智能（AGI）越来越多地融入到人类生活的各个方面，确保这些系统的安全和道德一致性至关重要。过去的研究主要关注单模态威胁，这可能不足以应对交叉模态相互作用的综合和复杂性。我们提出了一个名为 “安全输入但不安全输出（SIUO）” 的新型安全对齐挑战，以评估交叉模态的安全对齐性。具体而言，它考虑了单一模态在独立情况下安全，但在组合时可能导致不安全或不道德的输出的情况。为了对这个问题进行实证调查，我们开发了 SIUO，一个涵盖了 9 个关键安全领域（如自残、非法活动和隐私侵犯）的交叉模态基准。我们的研究结果揭示了闭源和开源 LVLM（如 GPT-4V 和 LLaVA）在安全性上存在重大漏洞，凸显了当前模型在可靠解读和应对复杂的现实情景方面的不足。

跨模态安全对齐

Cross-Modality Safety Alignment

We propose an efficient method to ground pretrained text-only language models
to the visual domain, enabling them to process and generate arbitrarily
interleaved image-and-text data. Our method leverages the abilities of language
models learnt from large scale text-only pretraining, such as in-context
learning and free-form text generation. We keep the language model frozen, and
finetune input and output linear layers to enable cross-modality interactions.
This allows our model to process arbitrarily interleaved image-and-text inputs,
and generate free-form text interleaved with retrieved images. We achieve
strong zero-shot performance on grounded tasks such as contextual image
retrieval and multimodal dialogue, and showcase compelling interactive
abilities. Our approach works with any off-the-shelf language model and paves
the way towards an effective, general solution for leveraging pretrained
language models in visually grounded settings.

该研究提出了一种有效的方法，将预训练的纯文本语言模型转移到视觉领域，使其能够处理和生成任意交错的图像和文本数据，并在上下文图像检索和多模态对话等方面实现了强有力的效果。

多模态生成：将语言模型与图像相结合

Grounding Language Models to Images for Multimodal Generation

Efficient video recognition is a hot-spot research topic with the explosive
growth of multimedia data on the Internet and mobile devices. Most existing
methods select the salient frames without awareness of the class-specific
saliency scores, which neglect the implicit association between the saliency of
frames and its belonging category. To alleviate this issue, we devise a novel
Temporal Saliency Query (TSQ) mechanism, which introduces class-specific
information to provide fine-grained cues for saliency measurement.
Specifically, we model the class-specific saliency measuring process as a
query-response task. For each category, the common pattern of it is employed as
a query and the most salient frames are responded to it. Then, the calculated
similarities are adopted as the frame saliency scores. To achieve it, we
propose a Temporal Saliency Query Network (TSQNet) that includes two
instantiations of the TSQ mechanism based on visual appearance similarities and
textual event-object relations. Afterward, cross-modality interactions are
imposed to promote the information exchange between them. Finally, we use the
class-specific saliencies of the most confident categories generated by two
modalities to perform the selection of salient frames. Extensive experiments
demonstrate the effectiveness of our method by achieving state-of-the-art
results on ActivityNet, FCVID and Mini-Kinetics datasets. Our project page is
at this https URL .

提出一种新的时序显著性查询机制，引入类别特定信息为显著性测量提供精细线索，并在 TSQNet 中实现了两种检索机制和跨模态交互，从而解决现有技术对类别的不确定性和上下文缺失等问题，并在 ActivityNet，FCVID 和 Mini-Kinetics 数据集上获得了最好的成果。