Recent advancements in diffusion-based video generation have showcased
remarkable results, yet the gap between synthetic and real-world videos remains
under-explored. In this study, we examine this gap from three fundamental
perspectives: appearance, motion, and geometry, comparing real-world videos
with those generated by a state-of-the-art AI model, Stable Video Diffusion. To
achieve this, we train three classifiers using 3D convolutional networks, each
targeting distinct aspects: vision foundation model features for appearance,
optical flow for motion, and monocular depth for geometry. Each classifier
exhibits strong performance in fake video detection, both qualitatively and
quantitatively. This indicates that AI-generated videos are still easily
detectable, and a significant gap between real and fake videos persists.
Furthermore, utilizing the Grad-CAM, we pinpoint systematic failures of
AI-generated videos in appearance, motion, and geometry. Finally, we propose an
Ensemble-of-Experts model that integrates appearance, optical flow, and depth
information for fake video detection, resulting in enhanced robustness and
generalization ability. Our model is capable of detecting videos generated by
Sora with high accuracy, even without exposure to any Sora videos during
training. This suggests that the gap between real and fake videos can be
generalized across various video generative models. Project page:
this https URL

通过实验分析外观、运动和几何等方面，在外观、运动和几何等方面对比了真实世界的视频和由先进 AI 模型生成的视频，揭示了真实世界视频与合成视频之间的差距，并提出了集成外观、光流和深度信息的专家集合模型以进行假视频检测，提高了鲁棒性和泛化能力。

检测像 Sora 这样的 AI 生成视频中的关键内容

What Matters in Detecting AI-Generated Videos like Sora?

Diffusion-based video generation has achieved significant progress, yet
generating multiple actions that occur sequentially remains a formidable task.
Directly generating a video with sequential actions can be extremely
challenging due to the scarcity of fine-grained action annotations and the
difficulty in establishing temporal semantic correspondences and maintaining
long-term consistency. To tackle this, we propose an intuitive and
straightforward solution: splicing multiple single-action video segments
sequentially. The core challenge lies in generating smooth and natural
transitions between these segments given the inherent complexity and
variability of action transitions. We introduce MAVIN (Multi-Action Video
INfilling model), designed to generate transition videos that seamlessly
connect two given videos, forming a cohesive integrated sequence. MAVIN
incorporates several innovative techniques to address challenges in the
transition video infilling task. Firstly, a consecutive noising strategy
coupled with variable-length sampling is employed to handle large infilling
gaps and varied generation lengths. Secondly, boundary frame guidance (BFG) is
proposed to address the lack of semantic guidance during transition generation.
Lastly, a Gaussian filter mixer (GFM) dynamically manages noise initialization
during inference, mitigating train-test discrepancy while preserving generation
flexibility. Additionally, we introduce a new metric, CLIP-RS (CLIP Relative
Smoothness), to evaluate temporal coherence and smoothness, complementing
traditional quality-based metrics. Experimental results on horse and tiger
scenarios demonstrate MAVIN's superior performance in generating smooth and
coherent video transitions compared to existing methods.

多动作视频过渡生成方法 MAVIN 在马和老虎场景中表现优于现有方法，生成流畅且连贯的视频过渡。

MAVIN：透过过渡视频填充的扩散模型实现多动作视频生成

MAVIN: Multi-Action Video Generation with Diffusion Models via  Transition Video Infilling

Though diffusion-based video generation has witnessed rapid progress, the
inference results of existing models still exhibit unsatisfactory temporal
consistency and unnatural dynamics. In this paper, we delve deep into the noise
initialization of video diffusion models, and discover an implicit
training-inference gap that attributes to the unsatisfactory inference quality.
Our key findings are: 1) the spatial-temporal frequency distribution of the
initial latent at inference is intrinsically different from that for training,
and 2) the denoising process is significantly influenced by the low-frequency
components of the initial noise. Motivated by these observations, we propose a
concise yet effective inference sampling strategy, FreeInit, which
significantly improves temporal consistency of videos generated by diffusion
models. Through iteratively refining the spatial-temporal low-frequency
components of the initial latent during inference, FreeInit is able to
compensate the initialization gap between training and inference, thus
effectively improving the subject appearance and temporal consistency of
generation results. Extensive experiments demonstrate that FreeInit
consistently enhances the generation results of various text-to-video
generation models without additional training.

通过深入研究视频扩散模型的噪声初始化，我们发现了一种训练与推理之间的隐式差距，为此我们提出了一种名为 FreeInit 的简洁而有效的推理采样策略，显著提高了扩散模型生成的视频的时间一致性。