In-context learning for vision data has been underexplored compared with that in natural language. Previous works studied image in-context learning, urging models to generate a single image guided by demonstrations. In this paper, we propose and study video in-context learning, where the model starts from an existing video clip and generates diverse potential future sequences, each semantically guided by the prompted video demonstrations. To achieve this, we provide a clear definition of the task, and train an autoregressive Transformer on video datasets. We thoroughly analyze the effect of different datasets and represent frames as discrete tokens, and then model them by next token predictions. We design various evaluation metrics, including both objective and subjective measures, to demonstrate the visual quality and semantic accuracy of generation results. Our model follows the scaling law and generates high-quality video clips that accurately align with the semantic guidance provided by in-context examples.

我们提出和研究了视频上下文学习，其中模型从现有视频片段开始生成多样的潜在未来序列，每个序列在语义上由提示的视频演示进行引导，并通过训练视频数据集上的自回归Transformer来模拟这个任务，通过各种评估指标展示了生成结果的视觉质量和语义准确性。