To reduce network traffic and support environments with limited resources, a
method for transmitting images with low amounts of transmission data is
required. Machine learning-based image compression methods, which compress the
data size of images while maintaining their features, have been proposed.
However, in certain situations, reconstructing a part of semantic information
of images at the receiver end may be sufficient. To realize this concept,
semantic-information-based communication, called semantic communication, has
been proposed, along with an image transmission method using semantic
communication. This method transmits only the semantic information of an image,
and the receiver reconstructs the image using an image-generation model. This
method utilizes one type of semantic information, but reconstructing images
similar to the original image using only it is challenging. This study proposes
a multi-modal image transmission method that leverages diverse semantic
information for efficient semantic communication. The proposed method extracts
multi-modal semantic information from an image and transmits only it.
Subsequently, the receiver generates multiple images using an image-generation
model and selects an output based on semantic similarity. The receiver must
select the output based only on the received features; however, evaluating
semantic similarity using conventional metrics is challenging. Therefore, this
study explored new metrics to evaluate the similarity between semantic features
of images and proposes two scoring procedures. The results indicate that the
proposed procedures can compare semantic similarities, such as position and
composition, between semantic features of the original and generated images.
Thus, the proposed method can facilitate the transmission and utilization of
photographs through mobile networks for various service applications.

该研究提出了一种基于多模态语义信息传输的图像传输方法，通过提取多模态语义信息并传输，接收方使用图像生成模型生成多张图像，并根据语义相似性进行选择。研究探索了评估原始图像和生成图像的语义特征相似性的新指标，并表明所提出的方法可促进通过移动网络进行各种服务应用的照片的传输与利用。

资源有限网络中基于多模态相似度估计的图像生成语义传播

Image Generative Semantic Communication with Multi-Modal Similarity  Estimation for Resource-Limited Networks

This paper introduces a deep learning approach to dynamic spectrum access,
leveraging the synergy of multi-modal image and spectrum data for the
identification of potential transmitters. We consider an edge device equipped
with a camera that is taking images of potential objects such as vehicles that
may harbor transmitters. Recognizing the computational constraints and trust
issues associated with on-device computation, we propose a collaborative system
wherein the edge device communicates selectively processed information to a
trusted receiver acting as a fusion center, where a decision is made to
identify whether a potential transmitter is present, or not. To achieve this,
we employ task-oriented communications, utilizing an encoder at the transmitter
for joint source coding, channel coding, and modulation. This architecture
efficiently transmits essential information of reduced dimension for object
classification. Simultaneously, the transmitted signals may reflect off objects
and return to the transmitter, allowing for the collection of target sensing
data. Then the collected sensing data undergoes a second round of encoding at
the transmitter, with the reduced-dimensional information communicated back to
the fusion center through task-oriented communications. On the receiver side, a
decoder performs the task of identifying a transmitter by fusing data received
through joint sensing and task-oriented communications. The two encoders at the
transmitter and the decoder at the receiver are jointly trained, enabling a
seamless integration of image classification and wireless signal detection.
Using AWGN and Rayleigh channel models, we demonstrate the effectiveness of the
proposed approach, showcasing high accuracy in transmitter identification
across diverse channel conditions while sustaining low latency in decision
making.

本研究介绍了一种深度学习方法，利用多模态图像和频谱数据相结合，实现动态频谱访问，并识别潜在的发射器。通过使用边缘设备上的摄像头拍摄潜在对象（例如可能携带发射器的车辆），我们提出了一种协作系统，其中边缘设备与可信接收器之间进行有选择的信息通信并作出决策，以确定潜在的发射器是否存在。通过任务导向的通信，在发射器上使用编码器进行联合源编码、信道编码和调制，从而高效传输用于对象分类的精简维度的重要信息。同时，发送的信号可能会反射到物体并返回到发射器，从而实现目标感知数据的收集。然后，在发射器上对收集到的感知数据进行第二轮编码，通过任务导向的通信将降维后的信息传回融合中心。在接收端，解码器通过联合感知和任务导向的通信，执行识别发射器的任务。发射器的两个编码器以及接收器的解码器进行联合训练，实现了图像分类和无线信号检测的无缝集成。通过 AWGN 和 Rayleigh 信道模型，我们展示了所提出方法的有效性，在各种信道条件下取得了发射器识别的高准确性，并保持低延迟的决策过程。