Current state-of-the-art approaches for spatio-temporal action localization
rely on detections at the frame level and model temporal context with 3D
ConvNets. Here, we go one step further and model spatio-temporal relations to
capture the interactions between human actors, relevant objects and scene
elements essential to differentiate similar human actions. Our approach is
weakly supervised and mines the relevant elements automatically with an
actor-centric relational network (ACRN). ACRN computes and accumulates
pair-wise relation information from actor and global scene features, and
generates relation features for action classification. It is implemented as
neural networks and can be trained jointly with an existing action detection
system. We show that ACRN outperforms alternative approaches which capture
relation information, and that the proposed framework improves upon the
state-of-the-art performance on JHMDB and AVA. A visualization of the learned
relation features confirms that our approach is able to attend to the relevant
relations for each action.

模型人与场景、物体之间的关系有助于精细化人类行为的区分，有别于现有 3D ConvNets 的方法，而采用 actor-centric relational network 以弱监督方式自动挖掘相关要素计算 pair-wise relation，实现 state-of-the-art 表现。