Spatio-temporal scene-graph approaches to video-based reasoning tasks, such
as video question-answering (QA), typically construct such graphs for every
video frame. These approaches often ignore the fact that videos are essentially
sequences of 2D "views" of events happening in a 3D space, and that the
semantics of the 3D scene can thus be carried over from frame to frame.
Leveraging this insight, we propose a (2.5+1)D scene graph representation to
better capture the spatio-temporal information flows inside the videos.
Specifically, we first create a 2.5D (pseudo-3D) scene graph by transforming
every 2D frame to have an inferred 3D structure using an off-the-shelf 2D-to-3D
transformation module, following which we register the video frames into a
shared (2.5+1)D spatio-temporal space and ground each 2D scene graph within it.
Such a (2.5+1)D graph is then segregated into a static sub-graph and a dynamic
sub-graph, corresponding to whether the objects within them usually move in the
world. The nodes in the dynamic graph are enriched with motion features
capturing their interactions with other graph nodes. Next, for the video QA
task, we present a novel transformer-based reasoning pipeline that embeds the
(2.5+1)D graph into a spatio-temporal hierarchical latent space, where the
sub-graphs and their interactions are captured at varied granularity. To
demonstrate the effectiveness of our approach, we present experiments on the
NExT-QA and AVSD-QA datasets. Our results show that our proposed (2.5+1)D
representation leads to faster training and inference, while our hierarchical
model showcases superior performance on the video QA task versus the state of
the art.

本文提出了一种基于 (2.5+1) D 场景图表示的视频问答方法，将视频帧转成伪 - 3D 视图并保持语义，然后基于此表示应用 transformer 模型进行推理，实验证明，该方法在视频问答任务中具有优异的性能。