This paper introduces StreamV2V, a diffusion model that achieves real-time
streaming video-to-video (V2V) translation with user prompts. Unlike prior V2V
methods using batches to process limited frames, we opt to process frames in a
streaming fashion, to support unlimited frames. At the heart of StreamV2V lies
a backward-looking principle that relates the present to the past. This is
realized by maintaining a feature bank, which archives information from past
frames. For incoming frames, StreamV2V extends self-attention to include banked
keys and values and directly fuses similar past features into the output. The
feature bank is continually updated by merging stored and new features, making
it compact but informative. StreamV2V stands out for its adaptability and
efficiency, seamlessly integrating with image diffusion models without
fine-tuning. It can run 20 FPS on one A100 GPU, being 15x, 46x, 108x, and 158x
faster than FlowVid, CoDeF, Rerender, and TokenFlow, respectively. Quantitative
metrics and user studies confirm StreamV2V's exceptional ability to maintain
temporal consistency.

StreamV2V 是一种实时的流媒体视频转视频（V2V）翻译方法，通过用户提示实现。它通过维护一个特征库，将过去的信息与当前帧进行融合，在处理无限帧数时，以流媒体方式而非批处理方式进行帧处理。StreamV2V 具有自适应性和高效性的特点，在不需要微调的情况下，与图像扩散模型无缝集成，并能在一个 A100 GPU 上以 20 FPS 的速度运行，比 FlowVid、CoDeF、Rerender 和 TokenFlow 分别快 15 倍、46 倍、108 倍和 158 倍。定量指标和用户研究结果证实了 StreamV2V 保持时间一致性的卓越能力。