Video tokenizers are essential for latent video diffusion models, converting raw video data into spatiotemporally compressed latent spaces for efficient training. However, extending state-of-the-art video tokenizers to achieve a temporal compression ratio beyond 4x without increasing channel capacity poses significant challenges. In this work, we propose an alternative approach to enhance temporal compression. We find that the reconstruction quality of temporally subsampled videos from a low-compression encoder surpasses that of high-compression encoders applied to original videos. This indicates that high-compression models can leverage representations from lower-compression models. Building on this insight, we develop a bootstrapped high-temporal-compression model that progressively trains high-compression blocks atop well-trained lower-compression models. Our method includes a cross-level feature-mixing module to retain information from the pretrained low-compression model and guide higher-compression blocks to capture the remaining details from the full video sequence. Evaluation of video benchmarks shows that our method significantly improves reconstruction quality while increasing temporal compression compared to direct extensions of existing video tokenizers. Furthermore, the resulting compact latent space effectively trains a video diffusion model for high-quality video generation with a reduced token budget.

本研究解决了在不增加通道容量的情况下，视频分词器实现超过4倍时间压缩比的挑战。通过发现低压缩编码器生成的时间下采样视频的重建质量优于高压缩编码器的原始视频重建，提出了一种新的渐进训练高压缩块的高时间压缩模型，显著提升了重建质量和时间压缩效率。这一方法对视频扩散模型的训练和高质量视频生成具有重要影响。

视频分词器的渐进式生长用于高压缩潜在空间