Grounded Situation Recognition (GSR) aims to generate structured semantic
summaries of images for "human-like" event understanding. Specifically, GSR
task not only detects the salient activity verb (e.g. buying), but also
predicts all corresponding semantic roles (e.g. agent and goods). Inspired by
object detection and image captioning tasks, existing methods typically employ
a two-stage framework: 1) detect the activity verb, and then 2) predict
semantic roles based on the detected verb. Obviously, this illogical framework
constitutes a huge obstacle to semantic understanding. First, pre-detecting
verbs solely without semantic roles inevitably fails to distinguish many
similar daily activities (e.g., offering and giving, buying and selling).
Second, predicting semantic roles in a closed auto-regressive manner can hardly
exploit the semantic relations among the verb and roles. To this end, in this
paper we propose a novel two-stage framework that focuses on utilizing such
bidirectional relations within verbs and roles. In the first stage, instead of
pre-detecting the verb, we postpone the detection step and assume a pseudo
label, where an intermediate representation for each corresponding semantic
role is learned from images. In the second stage, we exploit transformer layers
to unearth the potential semantic relations within both verbs and semantic
roles. With the help of a set of support images, an alternate learning scheme
is designed to simultaneously optimize the results: update the verb using nouns
corresponding to the image, and update nouns using verbs from support images.
Extensive experimental results on challenging SWiG benchmarks show that our
renovated framework outperforms other state-of-the-art methods under various
metrics.

本文提出了一种基于双向关系的新的两阶段框架，旨在利用动词和语义角色之间的双向关系，生成结构化的图像语义摘要，实现类人事件理解，实验结果表明该框架在挑战性的 SWiG 基准测试中优于其他最先进的方法。