Video-based visual relation detection tasks, such as video scene graph
generation, play important roles in fine-grained video understanding. However,
current video visual relation detection datasets have two main limitations that
hinder the progress of research in this area. First, they do not explore
complex human-human interactions in multi-person scenarios. Second, the
relation types of existing datasets have relatively low-level semantics and can
be often recognized by appearance or simple prior information, without the need
for detailed spatio-temporal context reasoning. Nevertheless, comprehending
high-level interactions between humans is crucial for understanding complex
multi-person videos, such as sports and surveillance videos. To address this
issue, we propose a new video visual relation detection task: video human-human
interaction detection, and build a dataset named SportsHHI for it. SportsHHI
contains 34 high-level interaction classes from basketball and volleyball
sports. 118,075 human bounding boxes and 50,649 interaction instances are
annotated on 11,398 keyframes. To benchmark this, we propose a two-stage
baseline method and conduct extensive experiments to reveal the key factors for
a successful human-human interaction detector. We hope that SportsHHI can
stimulate research on human interaction understanding in videos and promote the
development of spatio-temporal context modeling techniques in video visual
relation detection.

我们提出了一个新的视频视觉关系检测任务：视频人际互动检测，并构建了一个名为 SportsHHI 的数据集。该数据集包含来自篮球和排球运动的 34 个高级互动类别。我们希望 SportsHHI 能够刺激对视频中人际互动理解的研究，并促进视频视觉关系检测中关于时空上下文建模技术的发展。

SportsHHI：体育视频中人际互动检测数据集

SportsHHI: A Dataset for Human-Human Interaction Detection in Sports  Videos

Feature alignment is the primary means of fusing multimodal data. We propose
a feature alignment method that fully fuses multimodal information, which
alternately shifts and expands feature information from different modalities to
have a consistent representation in a feature space. The proposed method can
robustly capture high-level interactions between features of different
modalities, thus significantly improving the performance of multimodal
learning. We also show that the proposed method outperforms other popular
multimodal schemes on multiple tasks. Experimental evaluation of ETT and
MIT-BIH-Arrhythmia, datasets shows that the proposed method achieves state of
the art performance.

提出了一种特征对齐方法，该方法完全融合了多模态信息，可以稳健地捕捉不同模态特征之间的高级交互作用，从而显著提高多模态学习的性能，实验评估结果表明该方法达到了最先进的性能水平。