In action recognition, although the combination of spatio-temporal videos and
skeleton features can improve the recognition performance, a separate model and
balancing feature representation for cross-modal data are required. To solve
these problems, we propose Spatio-TemporAl cRoss (STAR)-transformer, which can
effectively represent two cross-modal features as a recognizable vector. First,
from the input video and skeleton sequence, video frames are output as global
grid tokens and skeletons are output as joint map tokens, respectively. These
tokens are then aggregated into multi-class tokens and input into
STAR-transformer. The STAR-transformer encoder layer consists of a full
self-attention (FAttn) module and a proposed zigzag spatio-temporal attention
(ZAttn) module. Similarly, the continuous decoder consists of a FAttn module
and a proposed binary spatio-temporal attention (BAttn) module.
STAR-transformer learns an efficient multi-feature representation of the
spatio-temporal features by properly arranging pairings of the FAttn, ZAttn,
and BAttn modules. Experimental results on the Penn-Action, NTU RGB+D 60, and
120 datasets show that the proposed method achieves a promising improvement in
performance in comparison to previous state-of-the-art methods.

提出一种名为 STAR-transformer 的模型，该模型可以有效地表示两个跨模态特征为可识别的矢量，并学习到一种高效的多功能特征表示方法。在动作识别领域中，STAR-transformer 的效果显著优于之前的最先进方法。

STAR-Transformer：面向人类动作识别的时空交叉注意力变压器

STAR-Transformer: A Spatio-temporal Cross Attention Transformer for Human Action Recognition

Although Transformer has achieved great successes on many NLP tasks, its
heavy structure with fully-connected attention connections leads to
dependencies on large training data. In this paper, we present
Star-Transformer, a lightweight alternative by careful sparsification. To
reduce model complexity, we replace the fully-connected structure with a
star-shaped topology, in which every two non-adjacent nodes are connected
through a shared relay node. Thus, complexity is reduced from quadratic to
linear, while preserving capacity to capture both local composition and
long-range dependency. The experiments on four tasks (22 datasets) show that
Star-Transformer achieved significant improvements against the standard
Transformer for the modestly sized datasets.

本文介绍了 Star-Transformer，一种轻量级的 NLP 模型，通过精细的稀疏化将全连接注意力连接结构替换为星形拓扑结构，将复杂性从二次降为线性，同时保持了捕获局部组合和长距离依赖性的能力，并在四个任务的 22 个数据集上取得了显著的性能提升。