Recent advances in internet-scale video data pretraining have led to the
development of text-to-video generative models that can create high-quality
videos across a broad range of visual concepts and styles. Due to their ability
to synthesize realistic motions and render complex objects, these generative
models have the potential to become general-purpose simulators of the physical
world. However, it is unclear how far we are from this goal with the existing
text-to-video generative models. To this end, we present VideoPhy, a benchmark
designed to assess whether the generated videos follow physical commonsense for
real-world activities (e.g. marbles will roll down when placed on a slanted
surface). Specifically, we curate a list of 688 captions that involve
interactions between various material types in the physical world (e.g.,
solid-solid, solid-fluid, fluid-fluid). We then generate videos conditioned on
these captions from diverse state-of-the-art text-to-video generative models,
including open models (e.g., VideoCrafter2) and closed models (e.g., Lumiere
from Google, Pika). Further, our human evaluation reveals that the existing
models severely lack the ability to generate videos adhering to the given text
prompts, while also lack physical commonsense. Specifically, the best
performing model, Pika, generates videos that adhere to the caption and
physical laws for only 19.7% of the instances. VideoPhy thus highlights that
the video generative models are far from accurately simulating the physical
world. Finally, we also supplement the dataset with an auto-evaluator,
VideoCon-Physics, to assess semantic adherence and physical commonsense at
scale.

通过创建一个基准测试集 VideoPhy，我们评估了现有的文本到视频生成模型是否能符合真实世界活动的物理常识，结果显示这些模型缺乏生成符合文字提示和物理规律视频的能力，从而揭示了视频生成模型远未准确模拟物理世界的程度。