We address the challenging problem of Natural Language Comprehension beyond
plain-text documents by introducing the TILT neural network architecture which
simultaneously learns layout information, visual features, and textual
semantics. Contrary to previous approaches, we rely on a decoder capable of
unifying a variety of problems involving natural language. The layout is
represented as an attention bias and complemented with contextualized visual
information, while the core of our model is a pretrained encoder-decoder
Transformer. Our novel approach achieves state-of-the-art results in extracting
information from documents and answering questions which demand layout
understanding (DocVQA, CORD, SROIE). At the same time, we simplify the process
by employing an end-to-end model.

本文介绍了 TILT 神经网络架构，该架构同时学习布局信息、视觉特征和文本语义的自然语言理解方式。与以往的方法不同，我们依赖于一个能够统一涉及自然语言的各种问题的解码器。该架构的核心是一个预训练的编码器 - 解码器 Transformer，其中布局表示为注意力偏差，并与上下文化的视觉信息相结合。我们的新方法在从文档中提取信息并回答需要布局理解的问题（如 DocVQA、CORD、SROIE）方面取得了最先进的结果。同时，我们通过采用端到端模型简化了该过程。