The Brazilian Supreme Court receives tens of thousands of cases each
semester. Court employees spend thousands of hours to execute the initial
analysis and classification of those cases -- which takes effort away from
posterior, more complex stages of the case management workflow. In this paper,
we explore multimodal classification of documents from Brazil's Supreme Court.
We train and evaluate our methods on a novel multimodal dataset of 6,510
lawsuits (339,478 pages) with manual annotation assigning each page to one of
six classes. Each lawsuit is an ordered sequence of pages, which are stored
both as an image and as a corresponding text extracted through optical
character recognition. We first train two unimodal classifiers: a ResNet
pre-trained on ImageNet is fine-tuned on the images, and a convolutional
network with filters of multiple kernel sizes is trained from scratch on
document texts. We use them as extractors of visual and textual features, which
are then combined through our proposed Fusion Module. Our Fusion Module can
handle missing textual or visual input by using learned embeddings for missing
data. Moreover, we experiment with bi-directional Long Short-Term Memory
(biLSTM) networks and linear-chain conditional random fields to model the
sequential nature of the pages. The multimodal approaches outperform both
textual and visual classifiers, especially when leveraging the sequential
nature of the pages.

本研究探讨了巴西最高法院文件的多模式分类，采用 ResNet 与多核卷积神经网络提取视觉和文本特征以及 biLSTM 网络建模页面的序列性，发现多模式分类优于单一模式分类。