Multimodal summarization usually suffers from the problem that the
contribution of the visual modality is unclear. Existing multimodal
summarization approaches focus on designing the fusion methods of different
modalities, while ignoring the adaptive conditions under which visual
modalities are useful. Therefore, we propose a novel Coarse-to-Fine
contribution network for multimodal Summarization (CFSum) to consider different
contributions of images for summarization. First, to eliminate the interference
of useless images, we propose a pre-filter module to abandon useless images.
Second, to make accurate use of useful images, we propose two levels of visual
complement modules, word level and phrase level. Specifically, image
contributions are calculated and are adopted to guide the attention of both
textual and visual modalities. Experimental results have shown that CFSum
significantly outperforms multiple strong baselines on the standard benchmark.
Furthermore, the analysis verifies that useful images can even help generate
non-visual words which are implicitly represented in the image.

我们提出了一种新的粗略到精细的多模态总结贡献网络 (CFSum)，以考虑图像在总结中的不同贡献。CFSum 在标准基准测试中明显优于多个强基线，并通过分析验证了图像对生成隐含在图像中的非视觉词语的帮助。