In this paper we introduce a new synchronisation task, Gesture-Sync:
determining if a person's gestures are correlated with their speech or not. In
comparison to Lip-Sync, Gesture-Sync is far more challenging as there is a far
looser relationship between the voice and body movement than there is between
voice and lip motion. We introduce a dual-encoder model for this task, and
compare a number of input representations including RGB frames, keypoint
images, and keypoint vectors, assessing their performance and advantages. We
show that the model can be trained using self-supervised learning alone, and
evaluate its performance on the LRS3 dataset. Finally, we demonstrate
applications of Gesture-Sync for audio-visual synchronisation, and in
determining who is the speaker in a crowd, without seeing their faces. The
code, datasets and pre-trained models can be found at:
https://www.robots.ox.ac.uk/~vgg/research/gestsync.

本文介绍了一项新的同步任务 Gesture-Sync：确定人的手势与他们的语言之间是否存在相关性。我们引入了一种双编码器模型进行这项任务，并比较了包括 RGB 帧、关键点图像和关键点向量在内的多种输入表示形式，评估了它们的性能和优势。我们展示了模型可以仅通过自监督学习进行训练，并在 LRS3 数据集上评估其性能。最后，我们展示了使用 Gesture-Sync 在音频 - 视觉同步和在人群中确定发言者的应用。代码、数据集和预训练模型可在以下网址找到：https://www.robots.ox.ac.uk/~vgg/research/gestsync。

GestSync：确定非言语角色的发言人

GestSync: Determining who is speaking without a talking head

We propose a cross-modal attention distillation framework to train a
dual-encoder model for vision-language understanding tasks, such as visual
reasoning and visual question answering. Dual-encoder models have a faster
inference speed than fusion-encoder models and enable the pre-computation of
images and text during inference. However, the shallow interaction module used
in dual-encoder models is insufficient to handle complex vision-language
understanding tasks. In order to learn deep interactions of images and text, we
introduce cross-modal attention distillation, which uses the image-to-text and
text-to-image attention distributions of a fusion-encoder model to guide the
training of our dual-encoder model. In addition, we show that applying the
cross-modal attention distillation for both pre-training and fine-tuning stages
achieves further improvements. Experimental results demonstrate that the
distilled dual-encoder model achieves competitive performance for visual
reasoning, visual entailment and visual question answering tasks while enjoying
a much faster inference speed than fusion-encoder models. Our code and models
will be publicly available at this https URL

提出了一种跨模态关注蒸馏框架来训练双编码器模型，以用于视觉语言理解任务，如视觉推理和视觉问答，并证明使用这种框架可以在保持比融合编码器模型更快的推理速度的同时实现竞争性的性能表现。

视觉语言理解的蒸馏双编码模型

Distilled Dual-Encoder Model for Vision-Language Understanding

We describe Verse by Verse, our experiment in augmenting the creative process
of writing poetry with an AI. We have created a group of AI poets, styled after
various American classic poets, that are able to offer as suggestions generated
lines of verse while a user is composing a poem. In this paper, we describe the
underlying system to offer these suggestions. This includes a generative model,
which is tasked with generating a large corpus of lines of verse offline and
which are then stored in an index, and a dual-encoder model that is tasked with
recommending the next possible set of verses from our index given the previous
line of verse.

本文介绍了我们使用人工智能增强写诗创作过程的实验 ——Verse by Verse。我们创建了一组样式各不相同的 AI 诗人，能够在用户创作诗歌时提供一些生成的诗句作为建议。文章描述了提供这些建议的基础系统，包括离线生成模型和双编码器模型。