Recent advances in text-to-image generative models provide the ability to
generate high-quality images from short text descriptions. These foundation
models, when pre-trained on billion-scale datasets, are effective for various
downstream tasks with little or no further training. A natural question to ask
is how such models may be adapted for image compression. We investigate several
techniques in which the pre-trained models can be directly used to implement
compression schemes targeting novel low rate regimes. We show how text
descriptions can be used in conjunction with side information to generate
high-fidelity reconstructions that preserve both semantics and spatial
structure of the original. We demonstrate that at very low bit-rates, our
method can significantly improve upon learned compressors in terms of
perceptual and semantic fidelity, despite no end-to-end training.

该论文研究了如何将基于 Billion-scale 数据集预训练的文本到图片生成模型用于图像压缩，并证明这些模型可以在很低的比特率情况下实现比学习压缩器更好的感知和语义保真度。

文本 + 草图：超低比特率图像压缩

Text + Sketch: Image Compression at Ultra Low Rates

This paper proposes a Perceptual Learned Video Compression (PLVC) approach
with recurrent conditional GAN. We employ the recurrent auto-encoder-based
compression network as the generator, and most importantly, we propose a
recurrent conditional discriminator, which judges on raw vs. compressed video
conditioned on both spatial and temporal features, including the latent
representation, temporal motion and hidden states in recurrent cells. This way,
the adversarial training pushes the generated video to be not only spatially
photo-realistic but also temporally consistent with the groundtruth and
coherent among video frames. The experimental results show that the learned
PLVC model compresses video with good perceptual quality at low bit-rate, and
that it outperforms the official HEVC test model (HM 16.20) and the existing
learned video compression approaches for several perceptual quality metrics and
user studies. The codes will be released at the project page:
this https URL

本文提出了一种适用于逐帧视频的感知式学习视频压缩 (Pervetual Learned Video Compression，PLVC) 方法，并首次提出了一种递归条件判别器，该方法可以在保留视频画质和压缩率同时，满足时域上的一致性，而且其性能远远好于现有的学习视频压缩算法及现有标准 HEVC HM 16.20。

感知学习视频压缩与循环条件 GAN

Perceptual Learned Video Compression with Recurrent Conditional GAN

The traditional image compressors, e.g., BPG and H.266, have achieved great
image and video compression quality. Recently, Convolutional Neural Network has
been used widely in image compression. We proposed an attention-based
convolutional neural network for low bit-rate compression to post-process the
output of traditional image compression decoder. Across the experimental
results on validation sets, the post-processing module trained by MAE and
MS-SSIM losses yields the highest PSNR of 32.10 on average at the bit-rate of
0.15.

提出了一种基于注意力机制的卷积神经网络用于低码率图像压缩的后处理，该后处理模块采用 MAE 和 MS-SSIM 损失训练，在 0.15 比特率下平均 PSNR 最高可达 32.10。