We propose a method for learning the posture and structure of agents from
unlabelled behavioral videos. Starting from the observation that behaving
agents are generally the main sources of movement in behavioral videos, our
method, Behavioral Keypoint Discovery (B-KinD), uses an encoder-decoder
architecture with a geometric bottleneck to reconstruct the spatiotemporal
difference between video frames. By focusing only on regions of movement, our
approach works directly on input videos without requiring manual annotations.
Experiments on a variety of agent types (mouse, fly, human, jellyfish, and
trees) demonstrate the generality of our approach and reveal that our
discovered keypoints represent semantically meaningful body parts, which
achieve state-of-the-art performance on keypoint regression among
self-supervised methods. Additionally, B-KinD achieve comparable performance to
supervised keypoints on downstream tasks, such as behavior classification,
suggesting that our method can dramatically reduce model training costs
vis-a-vis supervised methods.

该研究提出了一种基于 B-KinD 方法的自监督学习方法，使用编码器 - 解码器架构重建视频帧之间的时空差异，仅关注移动区域，探索语义意义的身体部位，可用于昂贵的监督学习替代方案。