Recent years have witnessed the rapid development of short videos, which
usually contain both visual and audio modalities. Background music is important
to the short videos, which can significantly influence the emotions of the
viewers. However, at present, the background music of short videos is generally
chosen by the video producer, and there is a lack of automatic music
recommendation methods for short videos. This paper introduces MVBind, an
innovative Music-Video embedding space Binding model for cross-modal retrieval.
MVBind operates as a self-supervised approach, acquiring inherent knowledge of
intermodal relationships directly from data, without the need of manual
annotations. Additionally, to compensate the lack of a corresponding
musical-visual pair dataset for short videos, we construct a dataset,
SVM-10K(Short Video with Music-10K), which mainly consists of meticulously
selected short videos. On this dataset, MVBind manifests significantly improved
performance compared to other baseline methods. The constructed dataset and
code will be released to facilitate future research.

该研究介绍了 MVBind，一种用于跨模态检索的创新音乐视频嵌入绑定模型。通过 MVBind，能够在没有手动注释的情况下，从数据中直接获取模态间关系的内在知识。此外，为了弥补短视频缺乏相应音乐 - 视觉对数据集的不足，该研究构建了 SVM-10K（具有精选的短视频和音乐的 10K 数据集），在该数据集上，MVBind 相比其他基准模型表现出显著提高。为方便未来研究，已经发布了构建的数据集和代码。

MVBIND: 基于嵌入空间绑定的自监督音乐视频推荐

MVBIND: Self-Supervised Music Recommendation For Videos Via Embedding  Space Binding

Nowadays, short videos (SVs) are essential to information acquisition and
sharing in our life. The prevailing use of SVs to spread emotions leads to the
necessity of emotion recognition in SVs. Considering the lack of SVs emotion
data, we introduce a large-scale dataset named eMotions, comprising 27,996
videos. Meanwhile, we alleviate the impact of subjectivities on labeling
quality by emphasizing better personnel allocations and multi-stage
annotations. In addition, we provide the category-balanced and test-oriented
variants through targeted data sampling. Some commonly used videos (e.g.,
facial expressions and postures) have been well studied. However, it is still
challenging to understand the emotions in SVs. Since the enhanced content
diversity brings more distinct semantic gaps and difficulties in learning
emotion-related features, and there exists information gaps caused by the
emotion incompleteness under the prevalently audio-visual co-expressions. To
tackle these problems, we present an end-to-end baseline method AV-CPNet that
employs the video transformer to better learn semantically relevant
representations. We further design the two-stage cross-modal fusion module to
complementarily model the correlations of audio-visual features. The EP-CE
Loss, incorporating three emotion polarities, is then applied to guide model
optimization. Extensive experimental results on nine datasets verify the
effectiveness of AV-CPNet. Datasets and code will be open on
this https URL

现今，短视频对于信息获取和分享至关重要。鉴于短视频情感数据的缺乏，我们介绍了一个大规模的数据集，名为 eMotions，包含 27996 个视频。我们通过优化人员分配和多阶段注释来减轻主观因素对标签质量的影响。另外，我们通过有针对性的数据采样提供了类别平衡和面向测试的变体。我们提出了一种端到端的基线方法 AV-CPNet，采用视频 Transformer 来更好地学习语义相关表示。我们进一步设计了两阶段的跨模态融合模块，来补充地建模音视频特征之间的相关性。然后，我们应用了包含三种情感极性的 EP-CE 损失来指导模型优化。对于九个数据集的广泛实验结果验证了 AV-CPNet 的有效性。数据集和代码将在此 https URL 开放。

eMotions：一份用于短视频情绪识别的大规模数据集

eMotions: A Large-Scale Dataset for Emotion Recognition in Short Videos

Short videos on social media are a prime way many young people find and
consume content. News outlets would like to reach audiences through news reels,
but currently struggle to translate traditional journalistic formats into the
short, entertaining videos that match the style of the platform. There are many
ways to frame a reel-style narrative around a news story, and selecting one is
a challenge. Different news stories call for different framings, and require a
different trade-off between entertainment and information. We present a system
called ReelFramer that uses text and image generation to help journalists
explore multiple narrative framings for a story, then generate scripts,
character boards and storyboards they can edit and iterate on. A user study of
five graduate students in journalism-related fields found the system greatly
eased the burden of transforming a written story into a reel, and that
exploring framings to find the right one was a rewarding process.

研究表明，通过使用文本和图像生成技术，ReelFramer 系统可以帮助新闻人士探索多种新闻故事的叙述框架，并生成脚本、角色板和故事板，以缓解将书面故事转化为短视频新闻的负担。