Generating Audio Description (AD) for movies is a challenging task that
requires fine-grained visual understanding and an awareness of the characters
and their names. Currently, visual language models for AD generation are
limited by a lack of suitable training data, and also their evaluation is
hampered by using performance measures not specialized to the AD domain. In
this paper, we make three contributions: (i) We propose two approaches for
constructing AD datasets with aligned video data, and build training and
evaluation datasets using these. These datasets will be publicly released; (ii)
We develop a Q-former-based architecture which ingests raw video and generates
AD, using frozen pre-trained visual encoders and large language models; and
(iii) We provide new evaluation metrics to benchmark AD quality that are
well-matched to human performance. Taken together, we improve the state of the
art on AD generation.

生成电影的音频描述（AD）是一项具有挑战性的任务，需要对细粒度的视觉理解和角色及其名称有意识。本文提出了两种构建与视频数据对齐的 AD 数据集的方法，并使用这些数据集构建了训练和评估数据集。我们还开发了基于 Q-former 的架构，它使用冻结的预训练视觉编码器和大型语言模型，输入原始视频并生成 AD。此外，我们还提供了新的评估指标来衡量 AD 质量，这些评估指标与人类表现相匹配，从而改进了 AD 生成的技术水平。