With the exponential growth of video traffic, traditional video streaming
systems are approaching their limits in compression efficiency and
communication capacity. To further reduce bitrate while maintaining quality, we
propose Promptus, a disruptive novel system that streaming prompts instead of
video content with Stable Diffusion, which converts video frames into a series
of "prompts" for delivery. To ensure pixel alignment, a gradient descent-based
prompt fitting framework is proposed. To achieve adaptive bitrate for prompts,
a low-rank decomposition-based bitrate control algorithm is introduced. For
inter-frame compression of prompts, a temporal smoothing-based prompt
interpolation algorithm is proposed. Evaluations across various video domains
and real network traces demonstrate Promptus can enhance the perceptual quality
by 0.111 and 0.092 (in LPIPS) compared to VAE and H.265, respectively, and
decreases the ratio of severely distorted frames by 89.3% and 91.7%. Moreover,
Promptus achieves real-time video generation from prompts at over 150 FPS. To
the best of our knowledge, Promptus is the first attempt to replace video
codecs with prompt inversion and the first to use prompt streaming instead of
video streaming. Our work opens up a new paradigm for efficient video
communication beyond the Shannon limit.

通过 Promptus，用稳定扩散将视频帧转换成一系列的 “prompts” 传递，可以进一步降低比特率并保持质量，其渐变下降的提示匹配框架、基于低秩分解的比特率控制算法以及基于时间平滑的提示插值算法可实现自适应比特率、帧压缩和实时视频生成，并提高感知质量，减少失真帧比例，开创了超越香农极限的高效视频通信新范式。

Promptus：可替代视频传输的稳定传播的提示流

Promptus: Can Prompts Streaming Replace Video Streaming with Stable  Diffusion

We propose a lossy image compression system using the deep-learning
autoencoder structure to participate in the Challenge on Learned Image
Compression (CLIC) 2018. Our autoencoder uses the residual blocks with skip
connections to reduce the correlation among image pixels and condense the input
image into a set of feature maps, a compact representation of the original
image. The bit allocation and bitrate control are implemented by using the
importance maps and quantizer. The importance maps are generated by a separate
neural net in the encoder. The autoencoder and the importance net are trained
jointly based on minimizing a weighted sum of mean squared error, MS-SSIM, and
a rate estimate. Our aim is to produce reconstructed images with good
subjective quality subject to the 0.15 bits-per-pixel constraint.

本文提出一种利用深度学习自编码器结构进行有损图像压缩的系统，通过残差块和跳跃连接降低图像像素相关性，使用重要性映射和量化器实现比特分配和比特率控制，该自编码器和重要性网络是联合训练，目的在于满足每像素 0.15 比特的限制条件下，产生具有良好主观质量的重构图像。