In this paper, we propose an integrated framework for multi-granular
explanation of video summarization. This framework integrates methods for
producing explanations both at the fragment level (indicating which video
fragments influenced the most the decisions of the summarizer) and the more
fine-grained visual object level (highlighting which visual objects were the
most influential for the summarizer). To build this framework, we extend our
previous work on this field, by investigating the use of a model-agnostic,
perturbation-based approach for fragment-level explanation of the video
summarization results, and introducing a new method that combines the results
of video panoptic segmentation with an adaptation of a perturbation-based
explanation approach to produce object-level explanations. The performance of
the developed framework is evaluated using a state-of-the-art summarization
method and two datasets for benchmarking video summarization. The findings of
the conducted quantitative and qualitative evaluations demonstrate the ability
of our framework to spot the most and least influential fragments and visual
objects of the video for the summarizer, and to provide a comprehensive set of
visual-based explanations about the output of the summarization process.

本论文提出了一个综合的视频摘要的多粒度解释框架，通过整合在片段级别和更细粒度的视觉对象级别产生解释的方法，通过使用模型无关的基于扰动的方法来在片段级解释视频摘要结果，并引入了将视频全景分割结果与基于扰动的解释方法相结合的新方法来产生对象级解释。我们对该框架进行了性能评估，使用一种最先进的摘要方法和两个用于基准测试视频摘要的数据集。定量和定性评估的结果表明，我们的框架能够发现视频摘要中最具影响力的片段和视觉对象，并提供了关于摘要过程输出的全面的基于视觉的解释。

视频总结的多粒度解释集成框架

An Integrated Framework for Multi-Granular Explanation of Video  Summarization

Vision-language pre-training has been an emerging and fast-developing
research topic, which transfers multi-modal knowledge from rich-resource
pre-training task to limited-resource downstream tasks. Unlike existing works
that predominantly learn a single generic encoder, we present a pre-trainable
Universal Encoder-DEcoder Network (Uni-EDEN) to facilitate both vision-language
perception (e.g., visual question answering) and generation (e.g., image
captioning). Uni-EDEN is a two-stream Transformer based structure, consisting
of three modules: object and sentence encoders that separately learns the
representations of each modality, and sentence decoder that enables both
multi-modal reasoning and sentence generation via inter-modal interaction.
Considering that the linguistic representations of each image can span
different granularities in this hierarchy including, from simple to
comprehensive, individual label, a phrase, and a natural sentence, we pre-train
Uni-EDEN through multi-granular vision-language proxy tasks: Masked Object
Classification (MOC), Masked Region Phrase Generation (MRPG), Image-Sentence
Matching (ISM), and Masked Sentence Generation (MSG). In this way, Uni-EDEN is
endowed with the power of both multi-modal representation extraction and
language modeling. Extensive experiments demonstrate the compelling
generalizability of Uni-EDEN by fine-tuning it to four vision-language
perception and generation downstream tasks.

本研究通过多颗粒度的视觉 - 语言代理任务对 Uni-EDEN 进行学习，以实现多模态表示提取和语言建模，并在 4 个视觉 - 语言知识下游任务中进行微调，表现出强大的泛化能力。