Action understanding matters and attracts attention. It can be formed as the
mapping from the action physical space to the semantic space. Typically,
researchers built action datasets according to idiosyncratic choices to define
classes and push the envelope of benchmarks respectively. Thus, datasets are
incompatible with each other like "Isolated Islands" due to semantic gaps and
various class granularities, e.g., do housework in dataset A and wash plate in
dataset B. We argue that a more principled semantic space is an urgent need to
concentrate the community efforts and enable us to use all datasets together to
pursue generalizable action learning. To this end, we design a Poincare action
semantic space given verb taxonomy hierarchy and covering massive actions. By
aligning the classes of previous datasets to our semantic space, we gather
(image/video/skeleton/MoCap) datasets into a unified database in a unified
label system, i.e., bridging "isolated islands" into a "Pangea". Accordingly,
we propose a bidirectional mapping model between physical and semantic space to
fully use Pangea. In extensive experiments, our system shows significant
superiority, especially in transfer learning. Code and data will be made
publicly available.

本文提出了一个基于 Poincare 动作语义空间的方法，将不相容的行动数据集转换为统一的数据库，并设计了一个双向映射模型，在实验中得到了显著更好的效果，特别是在迁移学习方面表现出卓越性。