Automatic assessment of the quality of scholarly documents is a difficult
task with high potential impact. Multimodality, in particular the addition of
visual information next to text, has been shown to improve the performance on
scholarly document quality prediction (SDQP) tasks. We propose the multimodal
predictive model MultiSChuBERT. It combines a textual model based on chunking
full paper text and aggregating computed BERT chunk-encodings (SChuBERT), with
a visual model based on Inception V3.Our work contributes to the current
state-of-the-art in SDQP in three ways. First, we show that the method of
combining visual and textual embeddings can substantially influence the
results. Second, we demonstrate that gradual-unfreezing of the weights of the
visual sub-model, reduces its tendency to ovefit the data, improving results.
Third, we show the retained benefit of multimodality when replacing standard
BERT$_{\textrm{BASE}}$ embeddings with more recent state-of-the-art text
embedding models.
Using BERT$_{\textrm{BASE}}$ embeddings, on the (log) number of citations
prediction task with the ACL-BiblioMetry dataset, our MultiSChuBERT
(text+visual) model obtains an $R^{2}$ score of 0.454 compared to 0.432 for the
SChuBERT (text only) model. Similar improvements are obtained on the PeerRead
accept/reject prediction task. In our experiments using SciBERT, scincl,
SPECTER and SPECTER2.0 embeddings, we show that each of these tailored
embeddings adds further improvements over the standard BERT$_{\textrm{BASE}}$
embeddings, with the SPECTER2.0 embeddings performing best.

多模态预测模型 MultiSChuBERT 通过使用基于 BERT 分块编码的文本模型（SChuBERT）和基于 Inception V3 的视觉模型，结合文本和可视信息，显著提高学术文档质量预测任务的结果。