The emerging video LMMs (Large Multimodal Models) have achieved significant
improvements on generic video understanding in the form of VQA (Visual Question
Answering), where the raw videos are captured by cameras. However, a large
portion of videos in real-world applications are edited videos, \textit{e.g.},
users usually cut and add effects/modifications to the raw video before
publishing it on social media platforms. The edited videos usually have high
view counts but they are not covered in existing benchmarks of video LMMs,
\textit{i.e.}, ActivityNet-QA, or VideoChatGPT benchmark. In this paper, we
leverage the edited videos on a popular short video platform, \textit{i.e.},
TikTok, and build a video VQA benchmark (named EditVid-QA) covering four
typical editing categories, i.e., effect, funny, meme, and game. Funny and meme
videos benchmark nuanced understanding and high-level reasoning, while effect
and game evaluate the understanding capability of artificial design. Most of
the open-source video LMMs perform poorly on the EditVid-QA benchmark,
indicating a huge domain gap between edited short videos on social media and
regular raw videos. To improve the generalization ability of LMMs, we collect a
training set for the proposed benchmark based on both Panda-70M/WebVid raw
videos and small-scale TikTok/CapCut edited videos, which boosts the
performance on the proposed EditVid-QA benchmark, indicating the effectiveness
of high-quality training data. We also identified a serious issue in the
existing evaluation protocol using the GPT-3.5 judge, namely a "sorry" attack,
where a sorry-style naive answer can achieve an extremely high rating from the
GPT judge, e.g., over 4.3 for correctness score on VideoChatGPT evaluation
protocol. To avoid the "sorry" attacks, we evaluate results with GPT-4 judge
and keyword filtering. The datasets will be released for academic purposes
only.

在本文中，我们针对社交媒体的编辑短视频构建了一个视频问答基准（称为 EditVid-QA），涵盖了四个典型的编辑类别，即特效、搞笑、网络迷因和游戏。我们的研究表明，现有的视频 LMMs 在编辑视频上表现较差，存在领域差距。为了提高 LMMs 的泛化能力，我们以 Panda-70M/WebVid 原始视频和小规模的 TikTok/CapCut 编辑视频为基础，收集了所提出基准的训练集，从而提升了在 EditVid-QA 基准上的性能。同时，我们还发现了现有评估协议中的一个严重问题，即使用 GPT-3.5 judge 的 “sorry” 攻击，为了避免这种攻击，我们使用了 GPT-4 judge 和关键词过滤来评估结果。该数据集仅供学术目的发布。