Automatic movie narration targets at creating video-aligned plot descriptions
to assist visually impaired audiences. It differs from standard video
captioning in that it requires not only describing key visual details but also
inferring the plots developed across multiple movie shots, thus posing unique
and ongoing challenges. To advance the development of automatic movie narrating
systems, we first revisit the limitations of existing datasets and develop a
large-scale, bilingual movie narration dataset, Movie101v2. Second, taking into
account the essential difficulties in achieving applicable movie narration, we
break the long-term goal into three progressive stages and tentatively focus on
the initial stages featuring understanding within individual clips. We also
introduce a new narration assessment to align with our staged task goals.
Third, using our new dataset, we baseline several leading large vision-language
models, including GPT-4V, and conduct in-depth investigations into the
challenges current models face for movie narration generation. Our findings
reveal that achieving applicable movie narration generation is a fascinating
goal that requires thorough research.

通过创建与视频对齐的情节描述来辅助视觉障碍观众，自动电影叙述与标准视频字幕不同，它需要描述关键的视觉细节，同时推断跨多个电影镜头发展的情节，因此存在独特和持续的挑战。为了推进自动电影叙述系统的发展，我们首先重新审视现有数据集的限制，并开发了一个大规模的双语电影叙述数据集 Movie101v2。其次，考虑到实现可应用的电影叙述的基本困难，我们将长期目标分为三个渐进阶段，并临时聚焦于特定片段内的理解。我们还引入了一种新的叙述评估来与我们的阶段性任务目标对齐。第三，利用我们的新数据集，我们对几种主要的大规模视觉语言模型进行了基准测试，包括 GPT-4V，并对当前模型在电影叙述生成方面面临的挑战进行了深入研究。我们的研究结果表明，实现可应用的电影叙述生成是一个引人入胜的目标，需要深入研究。

Movie101v2: 改进的电影叙事基准测试

Movie101v2: Improved Movie Narration Benchmark

Audio Description (AD) is the task of generating descriptions of visual
content, at suitable time intervals, for the benefit of visually impaired
audiences. For movies, this presents notable challenges -- AD must occur only
during existing pauses in dialogue, should refer to characters by name, and
ought to aid understanding of the storyline as a whole. To this end, we develop
a new model for automatically generating movie AD, given CLIP visual features
of the frames, the cast list, and the temporal locations of the speech;
addressing all three of the 'who', 'when', and 'what' questions: (i) who -- we
introduce a character bank consisting of the character's name, the actor that
played the part, and a CLIP feature of their face, for the principal cast of
each movie, and demonstrate how this can be used to improve naming in the
generated AD; (ii) when -- we investigate several models for determining
whether an AD should be generated for a time interval or not, based on the
visual content of the interval and its neighbours; and (iii) what -- we
implement a new vision-language model for this task, that can ingest the
proposals from the character bank, whilst conditioning on the visual features
using cross-attention, and demonstrate how this improves over previous
architectures for AD text generation in an apples-to-apples comparison.

为了解决电影音频描述中存在的挑战，如适应对话存在的间隙、通过角色名称引用以及整体上帮助理解剧情，我们开发了一个新模型来自动生成电影音频描述，使用了帧的 CLIP 视觉特征、演职员表和对话的时间位置，并解决了 ' 谁'、' 何时 ' 和' 什么 ' 三个问题，即通过引入角色库实现更好地命名、通过对时间间隔及其相邻内容的视觉特征进行模型选择以决定是否生成音频描述，以及通过视觉特征的交叉注意力在此任务中实现了一个新的视觉 - 语言模型，展示了与先前架构相比在音频描述生成方面的改进。