Is it possible to guess human action from dialogue alone? In this work we
investigate the link between spoken words and actions in movies. We note that
movie screenplays describe actions, as well as contain the speech of characters
and hence can be used to learn this correlation with no additional supervision.
We train a BERT-based Speech2Action classifier on over a thousand movie
screenplays, to predict action labels from transcribed speech segments. We then
apply this model to the speech segments of a large unlabelled movie corpus
(188M speech segments from 288K movies). Using the predictions of this model,
we obtain weak action labels for over 800K video clips. By training on these
video clips, we demonstrate superior action recognition performance on standard
action recognition benchmarks, without using a single manually labelled action
example.

本研究使用 BERT 基于对电影剧本的分析，研究了语音的文字转录以及行动之间的关联，训练了一个 Speech2Action 分类器，并在 188M 个片段的不带标签的电影语音数据上应用了该模型，获得了超过 800K 个视频片段的弱标签，最终通过使用这些标签标记的数据进行训练，在不使用单个手动标记行动示例的情况下，在标准动作识别基准测试上展现出卓越的行动识别表现。