To date, visual question answering (VQA) (i.e., image QA and video QA) is
still a holy grail in vision and language understanding, especially for video
QA. Compared with image QA that focuses primarily on understanding the
associations between image region-level details and corresponding questions,
video QA requires a model to jointly reason across both spatial and long-range
temporal structures of a video as well as text to provide an accurate answer.
In this paper, we specifically tackle the problem of video QA by proposing a
Structured Two-stream Attention network, namely STA, to answer a free-form or
open-ended natural language question about the content of a given video. First,
we infer rich long-range temporal structures in videos using our structured
segment component and encode text features. Then, our structured two-stream
attention component simultaneously localizes important visual instance, reduces
the influence of background video and focuses on the relevant text. Finally,
the structured two-stream fusion component incorporates different segments of
query and video aware context representation and infers the answers.
Experiments on the large-scale video QA dataset \textit{TGIF-QA} show that our
proposed method significantly surpasses the best counterpart (i.e., with one
representation for the video input) by 13.0%, 13.5%, 11.0% and 0.3 for Action,
Trans., TrameQA and Count tasks. It also outperforms the best competitor (i.e.,
with two representations) on the Action, Trans., TrameQA tasks by 4.1%, 4.7%,
and 5.1%.

本文提出了一种结构化的双流注意力网络（STA）来解决视频问答（VQA），该网络可以识别视频中的长时空结构和文本特征，并将视觉与文本融合以提供准确的答案，在大规模视频 QA 数据集 TGIF-QA 上实验表明，STA 可将 Action，Trans，TrameQA 和 Count 任务的最佳效果提高 13.0％，13.5％，11.0％和 0.3％，在 Action，Trans，TrameQA 任务上也比最佳竞争对手提高 4.1％，4.7％和 5.1％。