Manga, a widely celebrated Japanese comic art form, is renowned for its diverse narratives and distinct artistic styles. However, the inherently visual and intricate structure of Manga, which comprises images housing multiple panels, poses significant challenges for content retrieval. To address this, we present MaRU (Manga Retrieval and Understanding), a multi-staged system that connects vision and language to facilitate efficient search of both dialogues and scenes within Manga frames. The architecture of MaRU integrates an object detection model for identifying text and frame bounding boxes, a Vision Encoder-Decoder model for text recognition, a text encoder for embedding text, and a vision-text encoder that merges textual and visual information into a unified embedding space for scene retrieval. Rigorous evaluations reveal that MaRU excels in end-to-end dialogue retrieval and exhibits promising results for scene retrieval.

通过连接视觉和语言，我们提出了MaRU（Manga检索与理解），一个多阶段系统，旨在促进Manga帧内对话和场景的有效检索。MaRU的架构整合了文本和帧边界框的目标检测模型，用于文本识别的视觉编码-解码模型，用于嵌入文本的文本编码器，以及将文本和图像信息合并到统一嵌入空间以进行场景检索的视觉-文本编码器。严格的评估表明，MaRU在端到端对话检索方面表现出色，并对场景检索展现出有希望的结果。

MaRU：连接视觉与语言的漫画检索与理解系统