Video Question Answering (Video QA) is a powerful testbed to develop new AI
capabilities. This task necessitates learning to reason about objects,
relations, and events across visual and linguistic domains in space-time.
High-level reasoning demands lifting from associative visual pattern
recognition to symbol-like manipulation over objects, their behavior and
interactions. Toward reaching this goal we propose an object-oriented reasoning
approach in that video is abstracted as a dynamic stream of interacting
objects. At each stage of the video event flow, these objects interact with
each other, and their interactions are reasoned about with respect to the query
and under the overall context of a video. This mechanism is materialized into a
family of general-purpose neural units and their multi-level architecture
called Hierarchical Object-oriented Spatio-Temporal Reasoning (HOSTR) networks.
This neural model maintains the objects' consistent lifelines in the form of a
hierarchically nested spatio-temporal graph. Within this graph, the dynamic
interactive object-oriented representations are built up along the video
sequence, hierarchically abstracted in a bottom-up manner, and converge toward
the key information for the correct answer. The method is evaluated on multiple
major Video QA datasets and establishes new state-of-the-arts in these tasks.
Analysis into the model's behavior indicates that object-oriented reasoning is
a reliable, interpretable and efficient approach to Video QA.

通过提出一种基于对象导向推理的动态交互视频场景中的视觉物体关系、行为和事件的深度神经网络模型（HOSTR），能够在多个视频问题回答基准数据集中取得最新的最高成就并显示出其可靠性、可解释性和高效性。