We propose scene summarization as a new video-based scene understanding task.
It aims to summarize a long video walkthrough of a scene into a small set of
frames that are spatially diverse in the scene, which has many impotant
applications, such as in surveillance, real estate, and robotics. It stems from
video summarization but focuses on long and continuous videos from moving
cameras, instead of user-edited fragmented video clips that are more commonly
studied in existing video summarization works. Our solution to this task is a
two-stage self-supervised pipeline named SceneSum. Its first stage uses
clustering to segment the video sequence. Our key idea is to combine visual
place recognition (VPR) into this clustering process to promote spatial
diversity. Its second stage needs to select a representative keyframe from each
cluster as the summary while respecting resource constraints such as memory and
disk space limits. Additionally, if the ground truth image trajectory is
available, our method can be easily augmented with a supervised loss to enhance
the clustering and keyframe selection. Extensive experiments on both real-world
and simulated datasets show our method outperforms common video summarization
baselines by 50%

我们提出了场景摘要作为一项新的基于视频的场景理解任务，旨在将一个长视频场景的漫游摘要为一小组在场景中空间多样的帧，它有许多重要的应用场景，如监视、房地产和机器人。我们提出的解决方案是一个由两个阶段组成的自监督流程，名为 SceneSum，第一阶段使用聚类来分割视频序列，结合视觉地点识别（VPR）促进空间多样性，而第二阶段需要从每个簇中选择一个代表性关键帧作为摘要，同时考虑到资源限制，如内存和磁盘空间的限制，另外，如果有基准图像轨迹可用，我们的方法可以轻松增强聚类和关键帧选择的监督损失。在真实世界和模拟数据集上进行的大量实验证明，我们的方法的性能超过了普通视频摘要基线的 50%。