Video transition effects are widely used in video editing to connect shots
for creating cohesive and visually appealing videos. However, it is challenging
for non-professionals to choose best transitions due to the lack of
cinematographic knowledge and design skills. In this paper, we present the
premier work on performing automatic video transitions recommendation (VTR):
given a sequence of raw video shots and companion audio, recommend video
transitions for each pair of neighboring shots. To solve this task, we collect
a large-scale video transition dataset using publicly available video templates
on editing softwares. Then we formulate VTR as a multi-modal retrieval problem
from vision/audio to video transitions and propose a novel multi-modal matching
framework which consists of two parts. First we learn the embedding of video
transitions through a video transition classification task. Then we propose a
model to learn the matching correspondence from vision/audio inputs to video
transitions. Specifically, the proposed model employs a multi-modal transformer
to fuse vision and audio information, as well as capture the context cues in
sequential transition outputs. Through both quantitative and qualitative
experiments, we clearly demonstrate the effectiveness of our method. Notably,
in the comprehensive user study, our method receives comparable scores compared
with professional editors while improving the video editing efficiency by
\textbf{300\scalebox{1.25}{$\times$}}. We hope our work serves to inspire other
researchers to work on this new task. The dataset and codes are public at
https://github.com/acherstyx/AutoTransition.

本文介绍了一种基于多模态匹配的自动视频转场推荐方法，通过一个视频转场分类任务和一个匹配模型，该方法可以为每一组相邻的视频镜头推荐最佳的视频转场，得到了与专业编辑相当的用户评分，且可以提高视频编辑效率 300 倍。

自动过渡：学习推荐视频过渡效果

AutoTransition: Learning to Recommend Video Transition Effects

Image-text matching is an interesting and fascinating task in modern AI
research. Despite the evolution of deep-learning-based image and text
processing systems, multi-modal matching remains a challenging problem. In this
work, we consider the problem of accurate image-text matching for the task of
multi-modal large-scale information retrieval. State-of-the-art results in
image-text matching are achieved by inter-playing image and text features from
the two different processing pipelines, usually using mutual attention
mechanisms. However, this invalidates any chance to extract separate visual and
textual features needed for later indexing steps in large-scale retrieval
systems. In this regard, we introduce the Transformer Encoder Reasoning Network
(TERN), an architecture built upon one of the modern relationship-aware
self-attentive architectures, the Transformer Encoder (TE). This architecture
is able to separately reason on the two different modalities and to enforce a
final common abstract concept space by sharing the weights of the deeper
transformer layers. Thanks to this design, the implemented network is able to
produce compact and very rich visual and textual features available for the
successive indexing step. Experiments are conducted on the MS-COCO dataset, and
we evaluate the results using a discounted cumulative gain metric with
relevance computed exploiting caption similarities, in order to assess possibly
non-exact but relevant search results. We demonstrate that on this metric we
are able to achieve state-of-the-art results in the image retrieval task. Our
code is freely available at this https URL

本文介绍了使用 Transformer Encoder Reasoning Network 架构来进行图像与文本的自动匹配，以便于实现大规模信息检索。试验结果表明，本架构成功实现了该任务且获得了创新性的成果。