Scholarly articles in mathematical fields feature mathematical statements
such as theorems, propositions, etc., as well as their proofs. Extracting them
from the PDF representation of the articles requires understanding of
scientific text along with visual and font-based indicators. We pose this
problem as a multimodal classification problem using text, font features, and
bitmap image rendering of the PDF as different modalities. In this paper we
propose a multimodal machine learning approach for extraction of theorem-like
environments and proofs, based on late fusion of features extracted by
individual unimodal classifiers, taking into account the sequential succession
of blocks in the document. For the text modality, we pretrain a new language
model on a 11 GB scientific corpus; experiments shows similar performance for
our task than a model (RoBERTa) pretrained on 160 GB, with faster convergence
while requiring much less fine-tuning data. Font-based information relies on
training a 128-cell LSTM on the sequence of font names and sizes within each
block. Bitmap renderings are dealt with using an EfficientNetv2 deep network
tuned to classify each image block. Finally, a simple CRF-based approach uses
the features of the multimodal model along with information on block sequences.
Experimental results show the benefits of using a multimodal approach vs any
single modality, as well as major performance improvements using the CRF
modeling of block sequences.

使用多模态分类方法从数学文章中提取定理环境和证明，包括文本、字体信息和位图渲染等多个模态，并采用序列信息进行特征融合和模型训练。实验证明多模态方法比单一模态更有效，并且采用基于条件随机场的块序列建模可以显著提高性能。