The present paper proposes an encoder-decoder model for extracting the
structures of human motions represented by frame-wise discrete features in a
self-supervised manner. In the proposed method, features are extracted as codes
in a motion codebook without the use of human knowledge, and the relationship
between these codes can be visualized on a graph. Since the codes are expected
to be temporally sparse compared to the captured frame rate and can be shared
by multiple sequences, the proposed network model also addresses the need for
training constraints. Specifically, the model consists of self-attention layers
and a vector clustering block. The attention layers contribute to finding
sparse keyframes and discrete features as motion codes, which are then
extracted by vector clustering. The constraints are realized as training losses
so that the same motion codes can be as contiguous as possible and can be
shared by multiple sequences. In addition, we propose the use of causal
self-attention as a method by which to calculate attention for long sequences
consisting of numerous frames. In our experiments, the sparse structures of
motion codes were used to compile a graph that facilitates visualization of the
relationship between the codes and the differences between sequences. We then
evaluated the effectiveness of the extracted motion codes by applying them to
multiple recognition tasks and found that performance levels comparable to
task-optimized methods could be achieved by linear probing.

本文提出了一种编码器 - 解码器模型，用于自我监督方式提取以逐帧离散特征表示的人体动作的结构。

通过逐帧离散特征的自监督提取人体动作结构

Self-supervised Extraction of Human Motion Structures via Frame-wise  Discrete Features

Transformer-based language models have found many diverse applications
requiring them to process sequences of increasing length. For these
applications, the causal self-attention -- which is the only component scaling
quadratically w.r.t. the sequence length -- becomes a central concern. While
many works have proposed schemes to sparsify the attention patterns and reduce
the computational overhead of self-attention, those are often limited by
implementations concerns and end up imposing a simple and static structure over
the attention matrix. Conversely, implementing more dynamic sparse attentions
often results in runtimes significantly slower than computing the full
attention using the Flash implementation from Dao et al. (2022). We extend
FlashAttention to accommodate a large class of attention sparsity patterns
that, in particular, encompass key/query dropping and hashing-based attention.
This leads to implementations with no computational complexity overhead and a
multi-fold runtime speedup on top of FlashAttention. Even with relatively low
degrees of sparsity, our method improves visibly upon FlashAttention as the
sequence length increases. Without sacrificing perplexity, we increase the
training speed of a transformer language model by $2.0\times$ and $3.3\times$
for sequences of respectively $8k$ and $16k$ tokens.

我们扩展了 FlashAttention 以容纳一大类稀疏性注意力模式，其中包括关键 / 查询删除和基于哈希的注意力。即使具有相对较低的稀疏度，我们的方法在序列长度增加时可见地改进 FlashAttention 的性能。不牺牲困惑度的情况下，我们将 transformer 语言模型的训练速度分别提高了 2.0 倍和 3.3 倍，对于长度为 8k 和 16k 的序列。