This work explores a closure task in comics, a medium where visual and
textual elements are intricately intertwined. Specifically, Text-cloze refers
to the task of selecting the correct text to use in a comic panel, given its
neighboring panels. Traditional methods based on recurrent neural networks have
struggled with this task due to limited OCR accuracy and inherent model
limitations. We introduce a novel Multimodal Large Language Model
(Multimodal-LLM) architecture, specifically designed for Text-cloze, achieving
a 10% improvement over existing state-of-the-art models in both its easy and
hard variants. Central to our approach is a Domain-Adapted ResNet-50 based
visual encoder, fine-tuned to the comics domain in a self-supervised manner
using SimCLR. This encoder delivers comparable results to more complex models
with just one-fifth of the parameters. Additionally, we release new OCR
annotations for this dataset, enhancing model input quality and resulting in
another 1% improvement. Finally, we extend the task to a generative format,
establishing new baselines and expanding the research possibilities in the
field of comics analysis.

这项研究通过引入一种新颖的多模态大型语言模型（Multimodal-LLM）架构，特别设计用于文字片断选择的闭合任务（Text-cloze），并通过使用 SimCLR 以自监督的方式将 comics 领域进行了特定调整，以达到比现有最先进模型在易和难两种变种中提高了 10% 的效果。