Recently, the ever-increasing demand for bandwidth in multi-modal
communication systems requires a paradigm shift. Powered by deep learning,
semantic communications are applied to multi-modal scenarios to boost
communication efficiency and save communication resources. However, the
existing end-to-end neural network (NN) based framework without the channel
encoder/decoder is incompatible with modern digital communication systems.
Moreover, most end-to-end designs are task-specific and require re-design and
re-training for new tasks, which limits their applications. In this paper, we
propose a distributed multi-modal semantic communication framework
incorporating the conventional channel encoder/decoder. We adopt NN-based
semantic encoder and decoder to extract correlated semantic information
contained in different modalities, including speech, text, and image. Based on
the proposed framework, we further establish a general rate-adaptive coding
mechanism for various types of multi-modal semantic tasks. In particular, we
utilize unequal error protection based on semantic importance, which is derived
by evaluating the distortion bound of each modality. We further formulate and
solve an optimization problem that aims at minimizing inference delay while
maintaining inference accuracy for semantic tasks. Numerical results show that
the proposed mechanism fares better than both conventional communication and
existing semantic communication systems in terms of task performance, inference
delay, and deployment complexity.

本文提出一个利用神经网络解决多模态语义通信问题的分布式框架，该框架包括传统的信道编码和解码，进一步建立了针对各种多模态语义任务的一般速率自适应编码机制。优化方案旨在在保持语义任务推理准确性的同时，最小化推理延迟。本文的数值结果表明，所提出的机制总体上表现优于传统通信和现有的语义通信系统。