While progress has been made in the domain of video-language understanding,
current state-of-the-art algorithms are still limited in their ability to
understand videos at high levels of abstraction, such as news-oriented videos.
Alternatively, humans easily amalgamate information from video and language to
infer information beyond what is visually observable in the pixels. An example
of this is watching a news story, where the context of the event can play as
big of a role in understanding the story as the event itself. Towards a
solution for designing this ability in algorithms, we present a large-scale
analysis on an in-house dataset collected by the Reuters News Agency, called
Reuters Video-Language News (ReutersViLNews) dataset which focuses on
high-level video-language understanding with an emphasis on long-form news. The
ReutersViLNews Dataset consists of long-form news videos collected and labeled
by news industry professionals over several years and contains prominent news
reporting from around the world. Each video involves a single story and
contains action shots of the actual event, interviews with people associated
with the event, footage from nearby areas, and more. ReutersViLNews dataset
contains videos from seven subject categories: disaster, finance,
entertainment, health, politics, sports, and miscellaneous with annotations
from high-level to low-level, title caption, visual video description,
high-level story description, keywords, and location. We first present an
analysis of the dataset statistics of ReutersViLNews compared to previous
datasets. Then we benchmark state-of-the-art approaches for four different
video-language tasks. The results suggest that news-oriented videos are a
substantial challenge for current video-language understanding algorithms and
we conclude by providing future directions in designing approaches to solve the
ReutersViLNews dataset.

通过对 ReutersViLNews 数据集进行大规模分析，我们发现新闻导向的视频对当前视频语言理解算法构成了重大挑战，并提供了未来解决 ReutersViLNews 数据集的方法。

多模态新闻理解与专业标注视频

Multi-modal News Understanding with Professionally Labelled Videos  (ReutersViLNews)

A thorough comprehension of textual data is a fundamental element in
multi-modal video analysis tasks. However, recent works have shown that the
current models do not achieve a comprehensive understanding of the textual data
during the training for the target downstream tasks. Orthogonal to the previous
approaches to this limitation, we postulate that understanding the significance
of the sentence components according to the target task can potentially enhance
the performance of the models. Hence, we utilize the knowledge of a pre-trained
large language model (LLM) to generate text samples from the original ones,
targeting specific sentence components. We propose a weakly supervised
importance estimation module to compute the relative importance of the
components and utilize them to improve different video-language tasks. Through
rigorous quantitative analysis, our proposed method exhibits significant
improvement across several video-language tasks. In particular, our approach
notably enhances video-text retrieval by a relative improvement of 8.3\% in
video-to-text and 1.4\% in text-to-video retrieval over the baselines, in terms
of R@1. Additionally, in video moment retrieval, average mAP shows a relative
improvement ranging from 2.0\% to 13.7 \% across different baselines.

利用预训练的大型语言模型，通过计算句子组成部分的相对重要性并利用它们来改善不同的视频语言任务，我们的方法在多个视频语言任务中显著提升。