Despite great success has been achieved in activity analysis, it still has
many challenges. Most existing work in activity recognition pay more attention
to design efficient architecture or video sampling strategy. However, due to
the property of fine-grained action and long term structure in video, activity
recognition is expected to reason temporal relation between video sequences. In
this paper, we propose an efficient temporal reasoning graph (TRG) to
simultaneously capture the appearance features and temporal relation between
video sequences at multiple time scales. Specifically, we construct learnable
temporal relation graphs to explore temporal relation on the multi-scale range.
Additionally, to facilitate multi-scale temporal relation extraction, we design
a multi-head temporal adjacent matrix to represent multi-kinds of temporal
relations. Eventually, a multi-head temporal relation aggregator is proposed to
extract the semantic meaning of those features convolving through the graphs.
Extensive experiments are performed on widely-used large-scale datasets, such
as Something-Something and Charades, and the results show that our model can
achieve state-of-the-art performance. Further analysis shows that temporal
relation reasoning with our TRG can extract discriminative features for
activity recognition.

本文提出了一种高效的时间推理图 (TRG), 用于同时捕捉视频序列之间的外观特征和时间关系。通过构建可学习的时间关系图来探索多尺度范围上的时间关系。在多头时间邻接矩阵的帮助下，提出了一个多头时间关系聚合器来提取那些通过图卷积计算的特征的语义含义。实验证明，通过我们的 TRG 进行时间关系推理可以提取用于活动识别的判别特征。