Human Interaction Recognition is the process of identifying interactive
actions between multiple participants in a specific situation. The aim is to
recognise the action interactions between multiple entities and their meaning.
Many single Convolutional Neural Network has issues, such as the inability to
capture global instance interaction features or difficulty in training, leading
to ambiguity in action semantics. In addition, the computational complexity of
the Transformer cannot be ignored, and its ability to capture local information
and motion features in the image is poor. In this work, we propose a Two-stream
Hybrid CNN-Transformer Network (THCT-Net), which exploits the local specificity
of CNN and models global dependencies through the Transformer. CNN and
Transformer simultaneously model the entity, time and space relationships
between interactive entities respectively. Specifically, Transformer-based
stream integrates 3D convolutions with multi-head self-attention to learn
inter-token correlations; We propose a new multi-branch CNN framework for
CNN-based streams that automatically learns joint spatio-temporal features from
skeleton sequences. The convolutional layer independently learns the local
features of each joint neighborhood and aggregates the features of all joints.
And the raw skeleton coordinates as well as their temporal difference are
integrated with a dual-branch paradigm to fuse the motion features of the
skeleton. Besides, a residual structure is added to speed up training
convergence. Finally, the recognition results of the two branches are fused
using parallel splicing. Experimental results on diverse and challenging
datasets, demonstrate that the proposed method can better comprehend and infer
the meaning and context of various actions, outperforming state-of-the-art
methods.

我们提出了一种两流混合 CNN-Transformer 网络（THCT-Net），通过 CNN 和 Transformer 同时建模实体、时间和空间之间的关系，来更好地理解和推断各种动作的意义和上下文。

基于骨架的人体交互识别的双流混合 CNN-Transformer 网络

A Two-stream Hybrid CNN-Transformer Network for Skeleton-based Human  Interaction Recognition

Human interaction recognition is very important in many applications. One
crucial cue in recognizing an interaction is the interactive body parts. In
this work, we propose a novel Interaction Graph Transformer (IGFormer) network
for skeleton-based interaction recognition via modeling the interactive body
parts as graphs. More specifically, the proposed IGFormer constructs
interaction graphs according to the semantic and distance correlations between
the interactive body parts, and enhances the representation of each person by
aggregating the information of the interactive body parts based on the learned
graphs. Furthermore, we propose a Semantic Partition Module to transform each
human skeleton sequence into a Body-Part-Time sequence to better capture the
spatial and temporal information of the skeleton sequence for learning the
graphs. Extensive experiments on three benchmark datasets demonstrate that our
model outperforms the state-of-the-art with a significant margin.

本研究提出了一种新的基于骨架的交互作用识别模型 Interaction Graph Transformer (IGFormer), 通过将交互体部建模为图形来实现交互作用的识别，并通过学习的图形聚合交互体部的信息以增强每个人的表示，其中还提出了一个称为 Semantic Partition Module 的模块来更好地捕捉骨架序列的空间和时间信息以学习图形。实验证明，该模型优于现有技术。

IGFormer: 基于交互图转换器的基于骨架的人类交互识别

IGFormer: Interaction Graph Transformer for Skeleton-based Human  Interaction Recognition

In this paper, we aim to address the problem of human interaction recognition
in videos by exploring the long-term inter-related dynamics among multiple
persons. Recently, Long Short-Term Memory (LSTM) has become a popular choice to
model individual dynamic for single-person action recognition due to its
ability of capturing the temporal motion information in a range. However,
existing RNN models focus only on capturing the dynamics of human interaction
by simply combining all dynamics of individuals or modeling them as a whole.
Such models neglect the inter-related dynamics of how human interactions change
over time. To this end, we propose a novel Hierarchical Long Short-Term
Concurrent Memory (H-LSTCM) to model the long-term inter-related dynamics among
a group of persons for recognizing the human interactions. Specifically, we
first feed each person's static features into a Single-Person LSTM to learn the
single-person dynamic. Subsequently, the outputs of all Single-Person LSTM
units are fed into a novel Concurrent LSTM (Co-LSTM) unit, which mainly
consists of multiple sub-memory units, a new cell gate and a new co-memory
cell. In a Co-LSTM unit, each sub-memory unit stores individual motion
information, while this Co-LSTM unit selectively integrates and stores
inter-related motion information between multiple interacting persons from
multiple sub-memory units via the cell gate and co-memory cell, respectively.
Extensive experiments on four public datasets validate the effectiveness of the
proposed H-LSTCM by comparing against baseline and state-of-the-art methods.

本文提出了一种新型层级长短时记忆并发记忆模型（H-LSTCM），能够通过捕捉多个人之间的长期相互关系动态来识别人类交互，通过与基线和先进的方法相比，证明了所提出模型的有效性。