Multimodal summarisation with multimodal output is drawing increasing
attention due to the rapid growth of multimedia data. While several methods
have been proposed to summarise visual-text contents, their multimodal outputs
are not succinct enough at an extreme level to address the information overload
issue. To the end of extreme multimodal summarisation, we introduce a new task,
eXtreme Multimodal Summarisation with Multimodal Output (XMSMO) for the
scenario of TL;DW - Too Long; Didn't Watch, akin to TL;DR. XMSMO aims to
summarise a video-document pair into a summary with an extremely short length,
which consists of one cover frame as the visual summary and one sentence as the
textual summary. We propose a novel unsupervised Hierarchical Optimal Transport
Network (HOT-Net) consisting of three components: hierarchical multimodal
encoders, hierarchical multimodal fusion decoders, and optimal transport
solvers. Our method is trained, without using reference summaries, by
optimising the visual and textual coverage from the perspectives of the
distance between the semantic distributions under optimal transport plans. To
facilitate the study on this task, we collect a large-scale dataset XMSMO-News
by harvesting 4,891 video-document pairs. The experimental results show that
our method achieves promising performance in terms of ROUGE and IoU metrics.

介绍了一种新的极端多模式汇总方法（XMSMO），主要采用 HOT-Net 框架实现多模态输入到多模态输出的文本、图像总结，以解决信息过载问题。