Document Visual Question Answering (DocVQA) refers to the task of answering
questions from document images. Existing work on DocVQA only considers
single-page documents. However, in real scenarios documents are mostly composed
of multiple pages that should be processed altogether. In this work we extend
DocVQA to the multi-page scenario. For that, we first create a new dataset,
MP-DocVQA, where questions are posed over multi-page documents instead of
single pages. Second, we propose a new hierarchical method, Hi-VT5, based on
the T5 architecture, that overcomes the limitations of current methods to
process long multi-page documents. The proposed method is based on a
hierarchical transformer architecture where the encoder summarizes the most
relevant information of every page and then, the decoder takes this summarized
information to generate the final answer. Through extensive experimentation, we
demonstrate that our method is able, in a single stage, to answer the questions
and provide the page that contains the relevant information to find the answer,
which can be used as a kind of explainability measure.

本研究扩展了针对单页文档的 DocVQA 任务到多页文档的场景，提出了一个基于 T5 架构的新的分层方法 Hi-VT5，用于处理长的多页文档，并且通过实验证明本方法可以在单阶段完成回答问题和提供包含答案相关信息的页码，该信息可以作为可解释性量度。

多层次多模态 Transformer 用于多页文档问答任务

Hierarchical multimodal transformers for Multi-Page DocVQA

We investigate the performance of a state-of-the art (SoTA) architecture T5
(available on the SuperGLUE) and compare with it 3 other previous SoTA
architectures across 5 different tasks from 2 relatively diverse datasets. The
datasets are diverse in terms of the number and types of tasks they have. To
improve performance, we augment the training data by using an autoregressive
model. We achieve near-SoTA results on a couple of the tasks - macro F1 scores
of 81.66% for task A of the OLID 2019 dataset and 82.54% for task A of the hate
speech and offensive content (HASOC) 2021 dataset, where SoTA are 82.9% and
83.05%, respectively. We perform error analysis and explain why one of the
models (Bi-LSTM) makes the predictions it does by using a publicly available
algorithm: Integrated Gradient (IG). This is because explainable artificial
intelligence (XAI) is essential for earning the trust of users. The main
contributions of this work are the implementation method of T5, which is
discussed; the data augmentation using a new conversational AI model
checkpoint, which brought performance improvements; and the revelation on the
shortcomings of HASOC 2021 dataset. It reveals the difficulties of poor data
annotation by using a small set of examples where the T5 model made the correct
predictions, even when the ground truth of the test set were incorrect (in our
opinion). We also provide our model checkpoints on the HuggingFace hub1 to
foster transparency.

本研究使用 T5 架构及自回归模型对包含 5 个不同任务的 2 个数据集进行性能比较与错误分析，提出了一种新的数据增强方法以及训练方法并通过解释性人工智能算法提高了预测准确率。