We propose a new task of space-time semantic correspondence prediction in
videos. Given a source video, a target video, and a set of space-time
key-points in the source video, the task requires predicting a set of keypoints
in the target video that are the semantic correspondences of the provided
source keypoints. We believe that this task is important for fine-grain video
understanding, potentially enabling applications such as activity coaching,
sports analysis, robot imitation learning, and more. Our contributions in this
paper are: (i) proposing a new task and providing annotations for space-time
semantic correspondences on two existing benchmarks: Penn Action and Pouring;
and (ii) presenting a comprehensive set of baselines and experiments to gain
insights about the new problem. Our main finding is that the space-time
semantic correspondence prediction problem is best approached jointly in space
and time rather than in their decomposed sub-problems: time alignment and
spatial correspondences.

本论文提出了一个新的任务，即在视频中预测时空语义对应关系，为视频理解提供了新的思路，并且通过对两个现有基准数据集：Penn Action 和 Pouring 进行了注释和实验，最终得出了结论：最佳方法应综合考虑时空因素，而不是分别处理时间对齐和空间对应问题。