Document Visual Question Answering (DocVQA) refers to the task of answering
questions from document images. Existing work on DocVQA only considers
single-page documents. However, in real scenarios documents are mostly composed
of multiple pages that should be processed altogether. In this work we extend
DocVQA to the multi-page scenario. For that, we first create a new dataset,
MP-DocVQA, where questions are posed over multi-page documents instead of
single pages. Second, we propose a new hierarchical method, Hi-VT5, based on
the T5 architecture, that overcomes the limitations of current methods to
process long multi-page documents. The proposed method is based on a
hierarchical transformer architecture where the encoder summarizes the most
relevant information of every page and then, the decoder takes this summarized
information to generate the final answer. Through extensive experimentation, we
demonstrate that our method is able, in a single stage, to answer the questions
and provide the page that contains the relevant information to find the answer,
which can be used as a kind of explainability measure.

本研究扩展了针对单页文档的 DocVQA 任务到多页文档的场景，提出了一个基于 T5 架构的新的分层方法 Hi-VT5，用于处理长的多页文档，并且通过实验证明本方法可以在单阶段完成回答问题和提供包含答案相关信息的页码，该信息可以作为可解释性量度。

多层次多模态 Transformer 用于多页文档问答任务

Hierarchical multimodal transformers for Multi-Page DocVQA

Video understanding is emerging as a new paradigm for studying human-like AI.
Question-and-Answering (Q&A) is used as a general benchmark to measure the
level of intelligence for video understanding. While several previous studies
have suggested datasets for video Q&A tasks, they did not really incorporate
story-level understanding, resulting in highly-biased and lack of variance in
degree of question difficulty. In this paper, we propose a hierarchical method
for building Q&A datasets, i.e. hierarchical difficulty levels. We introduce
three criteria for video story understanding, i.e. memory capacity, logical
complexity, and DIKW (Data-Information-Knowledge-Wisdom) pyramid. We discuss
how three-dimensional map constructed from these criteria can be used as a
metric for evaluating the levels of intelligence relating to video story
understanding.

本文提出了一种分层方法来构建 Q&A 数据集，介绍了视频故事理解的三个标准：记忆容量、逻辑复杂度和 DIKW 金字塔，并讨论了如何从这些标准构建三维图来评估与视频故事理解相关的智能水平。

构建视频故事理解的分层问答数据集

Constructing Hierarchical Q&A Datasets for Video Story Understanding

In this paper we present a hierarchical method to discover mid-level elements
with the objective of modeling visual compatibility between objects. At the
base-level, our method identifies patterns of CNN activations with the aim of
modeling different variations/styles in which objects of the classes of
interest may occur. At the top-level, the proposed method discovers patterns of
co-occurring activations of base-level elements that define visual
compatibility between pairs of object classes. Experiments on the massive
Amazon dataset show the strength of our method at describing object classes and
the characteristics that drive the compatibility between them.

本文提出了一种分层方法，旨在建模对象之间的视觉兼容性，通过在基础级别识别 CNN 激活模式以建模目标类别的各种变化 / 风格，并在高级别上发现定义对象类别对之间视觉兼容性的基础级别元素的共同激活模式。在巨大的 Amazon 数据集上的实验表明了我们的方法在描述对象类别和驱动它们之间兼容性的特征方面的能力。