We propose an end-to-end image compression and analysis model with
Transformers, targeting to the cloud-based image classification application.
Instead of placing an existing Transformer-based image classification model
directly after an image codec, we aim to redesign the Vision Transformer (ViT)
model to perform image classification from the compressed features and
facilitate image compression with the long-term information from the
Transformer. Specifically, we first replace the patchify stem (i.e., image
splitting and embedding) of the ViT model with a lightweight image encoder
modelled by a convolutional neural network. The compressed features generated
by the image encoder are injected convolutional inductive bias and are fed to
the Transformer for image classification bypassing image reconstruction.
Meanwhile, we propose a feature aggregation module to fuse the compressed
features with the selected intermediate features of the Transformer, and feed
the aggregated features to a deconvolutional neural network for image
reconstruction. The aggregated features can obtain the long-term information
from the self-attention mechanism of the Transformer and improve the
compression performance. The rate-distortion-accuracy optimization problem is
finally solved by a two-step training strategy. Experimental results
demonstrate the effectiveness of the proposed model in both the image
compression and the classification tasks.

本文提出了一种基于 Transformer 的端到端图像压缩和分析模型，实现云端图像分类应用，并通过两步训练策略解决了率失真精度优化问题。实验结果表明，该模型在图像压缩和分类任务中均具有有效性。

基于 Transformer 的端到端图像压缩和分析

Towards End-to-End Image Compression and Analysis with Transformers

We present an end-to-end image compression system based on compressive
sensing. The presented system integrates the conventional scheme of compressive
sampling and reconstruction with quantization and entropy coding. The
compression performance, in terms of decoded image quality versus data rate, is
shown to be comparable with JPEG and significantly better at the low rate
range. We study the parameters that influence the system performance, including
(i) the choice of sensing matrix, (ii) the trade-off between quantization and
compression ratio, and (iii) the reconstruction algorithms. We propose an
effective method to jointly control the quantization step and compression ratio
in order to achieve near optimal quality at any given bit rate. Furthermore,
our proposed image compression system can be directly used in the compressive
sensing camera, e.g. the single pixel camera, to construct a hardware
compressive sampling system.

基于压缩感知的端到端图像压缩系统，结合传统压缩采样重建、量化和熵编码，可与 JPEG 相媲美，在低码率下表现显著优越。通过研究影响系统性能的参数，我们提出了一种有效的方法来联合控制量化步长和压缩比，以实现在任何给定比特率下达到接近最优的质量。此外，我们的压缩系统可直接用于压缩感知相机，例如单像素相机，构建一种硬件压缩采样系统。