Art forms such as movies and television (TV) dramas are reflections of the
real world, which have attracted much attention from the multimodal learning
community recently. However, existing corpora in this domain share three
limitations: (1) annotated in a scene-oriented fashion, they ignore the
coherence within plots; (2) their text lacks empathy and seldom mentions
situational context; (3) their video clips fail to cover long-form relationship
due to short duration. To address these fundamental issues, using 1,106 TV
drama episodes and 24,875 informative plot-focused sentences written by
professionals, with the help of 449 human annotators, we constructed PTVD, the
first plot-oriented multimodal dataset in the TV domain. It is also the first
non-English dataset of its kind. Additionally, PTVD contains more than 26
million bullet screen comments (BSCs), powering large-scale pre-training. Next,
aiming to open-source a strong baseline for follow-up works, we developed the
multimodal algorithm that attacks different cinema/TV modelling problems with a
unified architecture. Extensive experiments on three cognitive-inspired tasks
yielded a number of novel observations (some of them being quite
counter-intuition), further validating the value of PTVD in promoting
multimodal research. The dataset and codes are released at
https://ptvd.github.io/.

该研究通过使用 1,106 集电视剧和 24,875 个由专业人士编写的信息密集的剧情句子，进行 449 个人类注释者的标注，构建了第一个面向剧情的电视领域的多模态数据集 PTVD，同时也是该领域中第一个非英文的数据集，并开发了多模态算法，针对不同的电影 / 电视建模问题进行处理，并在三个认知启发任务上进行了广泛的实验，从而验证了该数据集在促进多模态研究上的价值。

基于电视剧的大规模情节导向多模态数据集 PTVD

PTVD: A Large-Scale Plot-Oriented Multimodal Dataset Based on Television  Dramas

We propose and investigate an identity sensitive joint embedding of face and
voice. Such an embedding enables cross-modal retrieval from voice to face and
from face to voice. We make the following four contributions: first, we show
that the embedding can be learnt from videos of talking faces, without
requiring any identity labels, using a form of cross-modal self-supervision;
second, we develop a curriculum learning schedule for hard negative mining
targeted to this task, that is essential for learning to proceed successfully;
third, we demonstrate and evaluate cross-modal retrieval for identities unseen
and unheard during training over a number of scenarios and establish a
benchmark for this novel task; finally, we show an application of using the
joint embedding for automatically retrieving and labelling characters in TV
dramas.

本文提出并研究了一种敏感于身份的面部和声音联合嵌入，该嵌入可以从声音到面部和从面部到声音进行交叉模态检索，并展示了使用该联合嵌入自动检索和标记电视剧角色的应用。