Visual contents, such as illustrations and images, play a big role in product
manual understanding. Existing Product Manual Question Answering (PMQA)
datasets tend to ignore visual contents and only retain textual parts. In this
work, to emphasize the importance of multimodal contents, we propose a
Multimodal Product Manual Question Answering (MPMQA) task. For each question,
MPMQA requires the model not only to process multimodal contents but also to
provide multimodal answers. To support MPMQA, a large-scale dataset PM209 is
constructed with human annotations, which contains 209 product manuals from 27
well-known consumer electronic brands. Human annotations include 6 types of
semantic regions for manual contents and 22,021 pairs of question and answer.
Especially, each answer consists of a textual sentence and related visual
regions from manuals. Taking into account the length of product manuals and the
fact that a question is always related to a small number of pages, MPMQA can be
naturally split into two subtasks: retrieving most related pages and then
generating multimodal answers. We further propose a unified model that can
perform these two subtasks all together and achieve comparable performance with
multiple task-specific models. The PM209 dataset is available at
this https URL

通过构建大规模的带有人工注释的 PM209 数据集，本研究提出了一个多模态产品手册问答（MPMQA）任务，其要求模型处理多模态内容并提供多模态答案，并进一步提出了一种统一模型，可以同时执行这两个子任务，并实现与多个任务特定模型相当的性能。