We introduce a new task -- language-driven video inpainting, which uses
natural language instructions to guide the inpainting process. This approach
overcomes the limitations of traditional video inpainting methods that depend
on manually labeled binary masks, a process often tedious and labor-intensive.
We present the Remove Objects from Videos by Instructions (ROVI) dataset,
containing 5,650 videos and 9,091 inpainting results, to support training and
evaluation for this task. We also propose a novel diffusion-based
language-driven video inpainting framework, the first end-to-end baseline for
this task, integrating Multimodal Large Language Models to understand and
execute complex language-based inpainting requests effectively. Our
comprehensive results showcase the dataset's versatility and the model's
effectiveness in various language-instructed inpainting scenarios. We will make
datasets, code, and models publicly available.

我们介绍了一种新的任务 - 语言驱动的视频修复，它使用自然语言指令来指导修复过程。我们提出了 Remove Objects from Videos by Instructions (ROVI) 数据集，其中包含 5,650 个视频和 9,091 个修复结果，以支持该任务的训练和评估。我们还提出了一种新的基于扩散的语言驱动视频修复框架，这是该任务的第一个端到端基线，有效地整合了多模态大规模语言模型，以理解和执行复杂的基于语言的修复请求。我们详细的结果展示了该数据集的多样性和该模型在各种语言指导下的修复场景中的有效性。我们将公开发布数据集、代码和模型。