With the evolution of storage and communication protocols, ultra-low bitrate
image compression has become a highly demanding topic. However, existing
compression algorithms must sacrifice either consistency with the ground truth
or perceptual quality at ultra-low bitrate. In recent years, the rapid
development of the Large Multimodal Model (LMM) has made it possible to balance
these two goals. To solve this problem, this paper proposes a method called
Multimodal Image Semantic Compression (MISC), which consists of an LMM encoder
for extracting the semantic information of the image, a map encoder to locate
the region corresponding to the semantic, an image encoder generates an
extremely compressed bitstream, and a decoder reconstructs the image based on
the above information. Experimental results show that our proposed MISC is
suitable for compressing both traditional Natural Sense Images (NSIs) and
emerging AI-Generated Images (AIGIs) content. It can achieve optimal
consistency and perception results while saving 50% bitrate, which has strong
potential applications in the next generation of storage and communication. The
code will be released on this https URL

该研究提出了一种名为多模态图像语义压缩（MISC）的方法，采用大型多模态模型（LMM）来平衡传统自然感知图像和人工智能生成图像的压缩，实现了一致性和感知结果的优化，节省了 50％的比特率，并在存储和通信领域具有强大的应用潜力。

MISC：基于大型多模态模型驱动的超低比特率图像语义压缩

MISC: Ultra-low Bitrate Image Semantic Compression Driven by Large  Multimodal Model

Image compression techniques typically focus on compressing rectangular
images for human consumption, however, resulting in transmitting redundant
content for downstream applications. To overcome this limitation, some previous
works propose to semantically structure the bitstream, which can meet specific
application requirements by selective transmission and reconstruction.
Nevertheless, they divide the input image into multiple rectangular regions
according to semantics and ignore avoiding information interaction among them,
causing waste of bitrate and distorted reconstruction of region boundaries. In
this paper, we propose to decouple an image into multiple groups with irregular
shapes based on a customized group mask and compress them independently. Our
group mask describes the image at a finer granularity, enabling significant
bitrate saving by reducing the transmission of redundant content. Moreover, to
ensure the fidelity of selective reconstruction, this paper proposes the
concept of group-independent transform that maintain the independence among
distinct groups. And we instantiate it by the proposed Group-Independent
Swin-Block (GI Swin-Block). Experimental results demonstrate that our framework
structures the bitstream with negligible cost, and exhibits superior
performance on both visual quality and intelligent task supporting.

本文提出了一种基于自定义组掩码将图像分解为多组不规则形状的方法，使得通过减少冗余内容的传输实现重大的比特率节省，并保证重建的可靠性。我们还提出了组独立变换的概念，通过提出的 GI Swin-Block 来实现，比现有技术表现出更好的性能。