In action recognition, although the combination of spatio-temporal videos and
skeleton features can improve the recognition performance, a separate model and
balancing feature representation for cross-modal data are required. To solve
these problems, we propose Spatio-TemporAl cRoss (STAR)-transformer, which can
effectively represent two cross-modal features as a recognizable vector. First,
from the input video and skeleton sequence, video frames are output as global
grid tokens and skeletons are output as joint map tokens, respectively. These
tokens are then aggregated into multi-class tokens and input into
STAR-transformer. The STAR-transformer encoder layer consists of a full
self-attention (FAttn) module and a proposed zigzag spatio-temporal attention
(ZAttn) module. Similarly, the continuous decoder consists of a FAttn module
and a proposed binary spatio-temporal attention (BAttn) module.
STAR-transformer learns an efficient multi-feature representation of the
spatio-temporal features by properly arranging pairings of the FAttn, ZAttn,
and BAttn modules. Experimental results on the Penn-Action, NTU RGB+D 60, and
120 datasets show that the proposed method achieves a promising improvement in
performance in comparison to previous state-of-the-art methods.

提出一种名为 STAR-transformer 的模型，该模型可以有效地表示两个跨模态特征为可识别的矢量，并学习到一种高效的多功能特征表示方法。在动作识别领域中，STAR-transformer 的效果显著优于之前的最先进方法。