A core process in human cognition is analogical mapping: the ability to
identify a similar relational structure between different situations. We
introduce a novel task, Visual Analogies of Situation Recognition, adapting the
classical word-analogy task into the visual domain. Given a triplet of images,
the task is to select an image candidate B' that completes the analogy (A to A'
is like B to what?). Unlike previous work on visual analogy that focused on
simple image transformations, we tackle complex analogies requiring
understanding of scenes.
We leverage situation recognition annotations and the CLIP model to generate
a large set of 500k candidate analogies. Crowdsourced annotations for a sample
of the data indicate that humans agree with the dataset label ~80% of the time
(chance level 25%). Furthermore, we use human annotations to create a
gold-standard dataset of 3,820 validated analogies. Our experiments demonstrate
that state-of-the-art models do well when distractors are chosen randomly
(~86%), but struggle with carefully chosen distractors (~53%, compared to 90%
human accuracy). We hope our dataset will encourage the development of new
analogy-making models. Website: this https URL

介绍了一项新颖的任务：视觉情境类比识别，通过 CLIP 模型生成了 50 万个类比数据集，并创建了一个 3820 个样本验证集，实验结果表明目前最先进的模型效果在随机干扰下良好，但在精心选择干扰时效果大打折扣，希望我们的数据集能鼓励新的类比制作模型的发展。

VASR: 情境识别的视觉类比

VASR: Visual Analogies of Situation Recognition

We propose a transfer deep learning (TDL) framework that can transfer the
knowledge obtained from a single-modal neural network to a network with a
different modality. Specifically, we show that we can leverage speech data to
fine-tune the network trained for video recognition, given an initial set of
audio-video parallel dataset within the same semantics. Our approach first
learns the analogy-preserving embeddings between the abstract representations
learned from intermediate layers of each network, allowing for semantics-level
transfer between the source and target modalities. We then apply our neural
network operation that fine-tunes the target network with the additional
knowledge transferred from the source network, while keeping the topology of
the target network unchanged. While we present an audio-visual recognition task
as an application of our approach, our framework is flexible and thus can work
with any multimodal dataset, or with any already-existing deep networks that
share the common underlying semantics. In this work in progress report, we aim
to provide comprehensive results of different configurations of the proposed
approach on two widely used audio-visual datasets, and we discuss potential
applications of the proposed approach.

这篇论文提出了一种转移学习框架，能将单一模态神经网络的知识迁移到另一个模态的网络上，通过语音数据微调已经训练过的视频识别网络，该方法首先学习抽象表示层中每个网络的类比映射，然后应用神经网络操作进行目标网络的微调，同时保持其拓扑不变，该框架能灵活地适用于任何多模态数据集或已存在的共享基础语义的深度网络。