The present paper proposes an encoder-decoder model for extracting the
structures of human motions represented by frame-wise discrete features in a
self-supervised manner. In the proposed method, features are extracted as codes
in a motion codebook without the use of human knowledge, and the relationship
between these codes can be visualized on a graph. Since the codes are expected
to be temporally sparse compared to the captured frame rate and can be shared
by multiple sequences, the proposed network model also addresses the need for
training constraints. Specifically, the model consists of self-attention layers
and a vector clustering block. The attention layers contribute to finding
sparse keyframes and discrete features as motion codes, which are then
extracted by vector clustering. The constraints are realized as training losses
so that the same motion codes can be as contiguous as possible and can be
shared by multiple sequences. In addition, we propose the use of causal
self-attention as a method by which to calculate attention for long sequences
consisting of numerous frames. In our experiments, the sparse structures of
motion codes were used to compile a graph that facilitates visualization of the
relationship between the codes and the differences between sequences. We then
evaluated the effectiveness of the extracted motion codes by applying them to
multiple recognition tasks and found that performance levels comparable to
task-optimized methods could be achieved by linear probing.

本文提出了一种编码器 - 解码器模型，用于自我监督方式提取以逐帧离散特征表示的人体动作的结构。