Several text-to-video diffusion models have demonstrated commendable
capabilities in synthesizing high-quality video content. However, it remains a
formidable challenge pertaining to maintaining temporal consistency and
ensuring action smoothness throughout the generated sequences. In this paper,
we present an innovative video generation AI agent that harnesses the power of
Sora-inspired multimodal learning to build skilled world models framework based
on textual prompts and accompanying images. The framework includes two parts:
prompt enhancer and full video translation. The first part employs the
capabilities of ChatGPT to meticulously distill and proactively construct
precise prompts for each subsequent step, thereby guaranteeing the utmost
accuracy in prompt communication and accurate execution in following model
operations. The second part employ compatible with existing advanced diffusion
techniques to expansively generate and refine the key frame at the conclusion
of a video. Then we can expertly harness the power of leading and trailing key
frames to craft videos with enhanced temporal consistency and action
smoothness. The experimental results confirm that our method has strong
effectiveness and novelty in constructing world models from text and image
inputs over the other methods.

通过使用基于 Sora 的多模态学习，利用文本提示和相关图像来构建熟练的世界模型框架，该方法在维护时间一致性和确保动作流畅性方面表现出强大的效果和创新性。