In light of the diminishing returns of traditional methods for enhancing
transmission rates, the domain of semantic communication presents promising new
frontiers. Focusing on image transmission, this paper explores the application
of foundation models, particularly the Segment Anything Model (SAM) developed
by Meta AI Research, to improve semantic communication. SAM is a promptable
image segmentation model that has gained attention for its ability to perform
zero-shot segmentation tasks without explicit training or domain-specific
knowledge. By employing SAM's segmentation capability and lightweight neural
network architecture for semantic coding, we propose a practical approach to
semantic communication. We demonstrate that this approach retains critical
semantic features, achieving higher image reconstruction quality and reducing
communication overhead. This practical solution eliminates the
resource-intensive stage of training a segmentation model and can be applied to
any semantic coding architecture, paving the way for real-world applications.

本文提出了一种基于 Segment Anything Model（SAM）的语义编码方法，SAM 作为一种可提示的图像分割模型，在不需要显示训练或具体领域知识的情况下能够完成零样本分割任务，通过应用 SAM 的分割能力和轻量级的神经网络架构进行语义编码，实现了对图像重建质量的提高和通信开销的降低。

分段大杂烩遇上语义交流

Segment Anything Meets Semantic Communication

Semantic communications has received growing interest since it can remarkably
reduce the amount of data to be transmitted without missing critical
information. Most existing works explore the semantic encoding and transmission
for text and apply techniques in Natural Language Processing (NLP) to interpret
the meaning of the text. In this paper, we conceive the semantic communications
for image data that is much more richer in semantics and bandwidth sensitive.
We propose an reinforcement learning based adaptive semantic coding (RL-ASC)
approach that encodes images beyond pixel level. Firstly, we define the
semantic concept of image data that includes the category, spatial arrangement,
and visual feature as the representation unit, and propose a convolutional
semantic encoder to extract semantic concepts. Secondly, we propose the image
reconstruction criterion that evolves from the traditional pixel similarity to
semantic similarity and perceptual performance. Thirdly, we design a novel
RL-based semantic bit allocation model, whose reward is the increase in
rate-semantic-perceptual performance after encoding a certain semantic concept
with adaptive quantization level. Thus, the task-related information is
preserved and reconstructed properly while less important data is discarded.
Finally, we propose the Generative Adversarial Nets (GANs) based semantic
decoder that fuses both locally and globally features via an attention module.
Experimental results demonstrate that the proposed RL-ASC is noise robust and
could reconstruct visually pleasant and semantic consistent image, and saves
times of bit cost compared to standard codecs and other deep learning-based
image codecs.

本文提出了一种基于强化学习的自适应语义编码方法 (RL-ASC)，通过卷积语义编码器将图像数据的类别、空间排列和视觉特征作为表征单元，将像素相似度替换为语义相似度来定义图像重建准则，通过自适应量化等级编码某个语义概念来增加速率 - 语义 - 感知性能作为奖励和生成式对抗网络 (GANs) 的语义解码器来重构图像，实验结果表明，与标准编解码器和其他深度学习基础图像编解码器相比，提出的 RL-ASC 方法具有噪声鲁棒性并能节约大量位量成本。