Natural language is leveraged in many computer vision tasks such as image
captioning, cross-modal retrieval or visual question answering, to provide
fine-grained semantic information. While human pose is key to human
understanding, current 3D human pose datasets lack detailed language
descriptions. In this work, we introduce the PoseScript dataset, which pairs a
few thousand 3D human poses from AMASS with rich human-annotated descriptions
of the body parts and their spatial relationships. To increase the size of this
dataset to a scale compatible with typical data hungry learning algorithms, we
propose an elaborate captioning process that generates automatic synthetic
descriptions in natural language from given 3D keypoints. This process extracts
low-level pose information -- the posecodes -- using a set of simple but
generic rules on the 3D keypoints. The posecodes are then combined into higher
level textual descriptions using syntactic rules. Automatic annotations
substantially increase the amount of available data, and make it possible to
effectively pretrain deep models for finetuning on human captions. To
demonstrate the potential of annotated poses, we show applications of the
PoseScript dataset to retrieval of relevant poses from large-scale datasets and
to synthetic pose generation, both based on a textual pose description.

本文提出了 PoseScript 数据集，该数据集对 AMASS 的几千个 3D 人体姿势与丰富的人体注释描述进行了配对。我们提出了一种详细的字幕生成过程，通过一组简单但通用的规则从给定的 3D 关键点提取低级姿势信息 —— 姿势码，然后使用句法规则将姿势码组合成更高级别的文本描述。自动注释大大增加了可用数据量，并使得在人类字幕微调预训练的深度模型上变得可能。