The goal of this paper is automatic character-aware subtitle generation.
Given a video and a minimal amount of metadata, we propose an audio-visual
method that generates a full transcript of the dialogue, with precise speech
timestamps, and the character speaking identified. The key idea is to first use
audio-visual cues to select a set of high-precision audio exemplars for each
character, and then use these exemplars to classify all speech segments by
speaker identity. Notably, the method does not require face detection or
tracking. We evaluate the method over a variety of TV sitcoms, including
Seinfeld, Fraiser and Scrubs. We envision this system being useful for the
automatic generation of subtitles to improve the accessibility of the vast
amount of videos available on modern streaming services. Project page :
https://www.robots.ox.ac.uk/~vgg/research/look-listen-recognise/

通过自动化、音频视觉的方法，本研究提出了一种生成具有精确语音时间戳和角色标识的对话全文本的字幕的方法，无需面部检测或追踪，旨在提高现代流媒体服务中大量视频的可访问性。