Understanding the human-object interactions (HOIs) from a video is essential
to fully comprehend a visual scene. This line of research has been addressed by
detecting HOIs from images and lately from videos. However, the video-based HOI
anticipation task in the third-person view remains understudied. In this paper,
we design a framework to detect current HOIs and anticipate future HOIs in
videos. We propose to leverage human gaze information since people often fixate
on an object before interacting with it. These gaze features together with the
scene contexts and the visual appearances of human-object pairs are fused
through a spatio-temporal transformer. To evaluate the model in the HOI
anticipation task in a multi-person scenario, we propose a set of person-wise
multi-label metrics. Our model is trained and validated on the VidHOI dataset,
which contains videos capturing daily life and is currently the largest video
HOI dataset. Experimental results in the HOI detection task show that our
approach improves the baseline by a great margin of 36.3% relatively. Moreover,
we conduct an extensive ablation study to demonstrate the effectiveness of our
modifications and extensions to the spatio-temporal transformer. Our code is
publicly available on this https URL

本文设计了一个框架来检测并预测视频中的人 - 物交互，并提出了使用人类注视信息、场景背景和人 - 物对的视觉外观等特征进行融合的时空变换器，并在 VidHOI 数据集上进行了模型训练和验证。