This paper presents a semi-supervised learning framework to train a keypoint
detector using multiview image streams given the limited labeled data
(typically $<$4\%). We leverage the complementary relationship between
multiview geometry and visual tracking to provide three types of supervisionary
signals to utilize the unlabeled data: (1) keypoint detection in one view can
be supervised by other views via the epipolar geometry; (2) a keypoint moves
smoothly over time where its optical flow can be used to temporally supervise
consecutive image frames to each other; (3) visible keypoint in one view is
likely to be visible in the adjacent view. We integrate these three signals in
a differentiable fashion to design a new end-to-end neural network composed of
three pathways. This design allows us to extensively use the unlabeled data to
train the keypoint detector. We show that our approach outperforms existing
detectors including DeepLabCut tailored to the keypoint detection of non-human
species such as monkeys, dogs, and mice.

本论文提出一个半监督学习框架，利用多视图图像流训练关键点检测器，以弥补有限标注数据的不足。作者使用多视图几何和视觉跟踪的互补关系提供三种监督信号，使用未标记的数据训练网络，证明其优于 DeepLabCut 等现有的关键点检测器，特别适用于猴、狗、老鼠等非人类物种的关键点检测。

多视角配准监督

Multiview Supervision By Registration

This paper presents MONET -- an end-to-end semi-supervised learning framework
for a keypoint detector using multiview image streams. In particular, we
consider general subjects such as non-human species where attaining a large
scale annotated dataset is challenging. While multiview geometry can be used to
self-supervise the unlabeled data, integrating the geometry into learning a
keypoint detector is challenging due to representation mismatch. We address
this mismatch by formulating a new differentiable representation of the
epipolar constraint called epipolar divergence---a generalized distance from
the epipolar lines to the corresponding keypoint distribution. Epipolar
divergence characterizes when two view keypoint distributions produce zero
reprojection error. We design a twin network that minimizes the epipolar
divergence through stereo rectification that can significantly alleviate
computational complexity and sampling aliasing in training. We demonstrate that
our framework can localize customized keypoints of diverse species, e.g.,
humans, dogs, and monkeys.

本文提出了一种半监督学习框架 MONET，用于使用多视图图像流检测关键点，在具有挑战性的获取大规模标注数据的非人类物种中，我们考虑使用多视图几何可用于自监督非标记数据，通过制定称为极线离散度的新的可微分表示来解决表示不匹配，并使用立体校正来最小化极线离散度，以便能够本文演示的多种物种的关键点定位。