We present a general approach to video understanding, inspired by semantic
transfer techniques that have been successfully used for 2D image analysis. Our
method considers a video to be a 1D sequence of clips, each one associated with
its own semantics. The nature of these semantics -- natural language captions
or other labels -- depends on the task at hand. A test video is processed by
forming correspondences between its clips and the clips of reference videos
with known semantics, following which, reference semantics can be transferred
to the test video. We describe two matching methods, both designed to ensure
that (a) reference clips appear similar to test clips and (b), taken together,
the semantics of the selected reference clips is consistent and maintains
temporal coherence. We use our method for video captioning on the LSMDC'16
benchmark, video summarization on the SumMe and TVSum benchmarks, Temporal
Action Detection on the Thumos2014 benchmark, and sound prediction on the
Greatest Hits benchmark. Our method not only surpasses the state of the art, in
four out of five benchmarks, but importantly, it is the only single method we
know of that was successfully applied to such a diverse range of tasks.

本论文提出了一种通用且灵感来源于对 2D 图像分析成功的语义转移技术的视频理解方法，通过测试视频和具有已知语义参考视频之间的对应关系，将参考的语义转移到测试视频。我们讨论了两种匹配方法，并将其应用于视频字幕，视频摘要，时间动作检测和声音预测等任务，并在 LSMDC'16 benchmark，SumMe 和 TVSum benchmarks，Thumos2014 benchmark 和 Greatest Hits benchmark 上取得了优异的表现。