This paper introduces StreamV2V, a diffusion model that achieves real-time
streaming video-to-video (V2V) translation with user prompts. Unlike prior V2V
methods using batches to process limited frames, we opt to process frames in a
streaming fashion, to support unlimited frames. At the heart of StreamV2V lies
a backward-looking principle that relates the present to the past. This is
realized by maintaining a feature bank, which archives information from past
frames. For incoming frames, StreamV2V extends self-attention to include banked
keys and values and directly fuses similar past features into the output. The
feature bank is continually updated by merging stored and new features, making
it compact but informative. StreamV2V stands out for its adaptability and
efficiency, seamlessly integrating with image diffusion models without
fine-tuning. It can run 20 FPS on one A100 GPU, being 15x, 46x, 108x, and 158x
faster than FlowVid, CoDeF, Rerender, and TokenFlow, respectively. Quantitative
metrics and user studies confirm StreamV2V's exceptional ability to maintain
temporal consistency.

StreamV2V 是一种实时的流媒体视频转视频（V2V）翻译方法，通过用户提示实现。它通过维护一个特征库，将过去的信息与当前帧进行融合，在处理无限帧数时，以流媒体方式而非批处理方式进行帧处理。StreamV2V 具有自适应性和高效性的特点，在不需要微调的情况下，与图像扩散模型无缝集成，并能在一个 A100 GPU 上以 20 FPS 的速度运行，比 FlowVid、CoDeF、Rerender 和 TokenFlow 分别快 15 倍、46 倍、108 倍和 158 倍。定量指标和用户研究结果证实了 StreamV2V 保持时间一致性的卓越能力。

回顾：具备特征存储的视频到视频翻译技术

Looking Backward: Streaming Video-to-Video Translation with Feature  Banks

Learning effective spectral-spatial features is important for the
hyperspectral image (HSI) classification task, but the majority of existing HSI
classification methods still suffer from modeling complex spectral-spatial
relations and characterizing low-level details and high-level semantics
comprehensively. As a new class of record-breaking generative models, diffusion
models are capable of modeling complex relations for understanding inputs well
as learning both high-level and low-level visual features. Meanwhile, diffusion
models can capture more abundant features by taking advantage of the extra and
unique dimension of timestep t. In view of these, we propose an unsupervised
spectral-spatial feature learning framework based on the diffusion model for
HSI classification for the first time, named Diff-HSI. Specifically, we first
pretrain the diffusion model with unlabeled HSI patches for unsupervised
feature learning, and then exploit intermediate hierarchical features from
different timesteps for classification. For better using the abundant
timestep-wise features, we design a timestep-wise feature bank and a dynamic
feature fusion module to construct timestep-wise features, adaptively learning
informative multi-timestep representations. Finally, an ensemble of linear
classifiers is applied to perform HSI classification. Extensive experiments are
conducted on three public HSI datasets, and our results demonstrate that
Diff-HSI outperforms state-of-the-art supervised and unsupervised methods for
HSI classification.

本文提出了一种基于扩散模型的非监督超光谱图像 (HSI) 特征学习框架 Diff-HSI，其可以利用时间步长性质捕捉更丰富的特征，设计了一个时间步骤的特征库和动态特征融合模块，最后实现了 HSI 分类。在三个公共 HSI 数据集上进行的广泛实验证明，Diff-HSI 优于各种现有的监督和非监督 HSI 分类方法。