We present EgoExo-Fitness, a new full-body action understanding dataset,
featuring fitness sequence videos recorded from synchronized egocentric and
fixed exocentric (third-person) cameras. Compared with existing full-body
action understanding datasets, EgoExo-Fitness not only contains videos from
first-person perspectives, but also provides rich annotations. Specifically,
two-level temporal boundaries are provided to localize single action videos
along with sub-steps of each action. More importantly, EgoExo-Fitness
introduces innovative annotations for interpretable action judgement--including
technical keypoint verification, natural language comments on action execution,
and action quality scores. Combining all of these, EgoExo-Fitness provides new
resources to study egocentric and exocentric full-body action understanding
across dimensions of "what", "when", and "how well". To facilitate research on
egocentric and exocentric full-body action understanding, we construct
benchmarks on a suite of tasks (i.e., action classification, action
localization, cross-view sequence verification, cross-view skill determination,
and a newly proposed task of guidance-based execution verification), together
with detailed analysis. Code and data will be available at
this https URL

我们呈现了 EgoExo-Fitness，这是一个全新的全身动作理解数据集，通过同步的自主和固定的第三人称摄像机记录了健身序列视频。与现有的全身动作理解数据集相比，EgoExo-Fitness 不仅包含第一人称视角的视频，还提供了丰富的注释，包括单个动作视频的两级时间界限以及每个动作的子步骤等。重要的是，EgoExo-Fitness 引入了新颖的注释，包括技术关键点验证、对动作执行的自然语言评论以及动作质量评分，为研究自主和固定视角下全身动作理解提供了新的资源。为了促进对自主和固定视角下全身动作理解的研究，我们构建了一套任务基准（即动作分类、动作定位、跨视图序列验证、跨视图技能判断和新提出的基于指导的执行验证任务），并进行了详细分析。代码和数据将在此 https URL 提供。

EgoExo-Fitness: 走向自视角和他视角的全身动作理解

EgoExo-Fitness: Towards Egocentric and Exocentric Full-Body Action  Understanding

First-person (egocentric) and third person (exocentric) videos are
drastically different in nature. The relationship between these two views have
been studied in recent years, however, it has yet to be fully explored. In this
work, we introduce two datasets (synthetic and natural/real) containing
simultaneously recorded egocentric and exocentric videos. We also explore
relating the two domains (egocentric and exocentric) in two aspects. First, we
synthesize images in the egocentric domain from the exocentric domain using a
conditional generative adversarial network (cGAN). We show that with enough
training data, our network is capable of hallucinating how the world would look
like from an egocentric perspective, given an exocentric video. Second, we
address the cross-view retrieval problem across the two views. Given an
egocentric query frame (or its momentary optical flow), we retrieve its
corresponding exocentric frame (or optical flow) from a gallery set. We show
that using synthetic data could be beneficial in retrieving real data. We show
that performing domain adaptation from the synthetic domain to the natural/real
domain, is helpful in tasks such as retrieval. We believe that the presented
datasets and the proposed baselines offer new opportunities for further
research in this direction. The code and dataset are publicly available.

本研究介绍了包含同时录制自我中心和外部中心视频的两个数据集（合成和自然 / 真实），首先使用条件生成对抗网络从外部中心领域合成自我中心领域的图像。 其次，我们解决了跨视图检索问题，并通过从合成域到自然 / 真实域的域适应来提高性能。