Large language models (LLMs) have significantly advanced audio processing
through audio codecs that convert audio into discrete tokens, enabling the
application of language modelling techniques to audio data. However,
traditional codecs often operate at high bitrates or within narrow domains such
as speech and lack the semantic clues required for efficient language
modelling. Addressing these challenges, we introduce SemantiCodec, a novel
codec designed to compress audio into fewer than a hundred tokens per second
across diverse audio types, including speech, general audio, and music, without
compromising quality. SemantiCodec features a dual-encoder architecture: a
semantic encoder using a self-supervised AudioMAE, discretized using k-means
clustering on extensive audio data, and an acoustic encoder to capture the
remaining details. The semantic and acoustic encoder outputs are used to
reconstruct audio via a diffusion-model-based decoder. SemantiCodec is
presented in three variants with token rates of 25, 50, and 100 per second,
supporting a range of ultra-low bit rates between 0.31 kbps and 1.43 kbps.
Experimental results demonstrate that SemantiCodec significantly outperforms
the state-of-the-art Descript codec on reconstruction quality. Our results also
suggest that SemantiCodec contains significantly richer semantic information
than all evaluated audio codecs, even at significantly lower bitrates. Our code
and demos are available at this https URL

利用语言模型和音频编解码技术，SemantiCodec 是一种能够以非常低的比特率将各种类型的音频压缩为每秒少于一百个标记的新型编解码器，具有更高的重构质量和更丰富的语义信息。

SemantiCodec：一种用于通用声音的超低比特率语义音频编解码器

SemantiCodec: An Ultra Low Bitrate Semantic Audio Codec for General  Sound

In this paper, we have expanded the current status of semantic communication
limited to processing one task to a more general system that can handle
multiple tasks concurrently. In pursuit of this, we first introduced our
definition of the "semantic source", enabling the interpretation of multiple
semantics based on a single observation. A semantic encoder design is then
introduced, featuring the division of the encoder into a common unit and
multiple specific units enabling cooperative multi-task processing. Simulation
results demonstrate the effectiveness of the proposed semantic source and the
system design. Our approach employs information maximization (infomax) and
end-to-end design principles.

我们扩展了当前语义通信的状况，使其能够同时处理多个任务。我们首先介绍了我们关于 “语义源” 的定义，其能够基于单个观察解释多个语义。然后介绍了一个语义编码器设计，其中编码器分为通用单元和多个特定单元，实现了协同多任务处理。模拟结果证明了提出的语义源和系统设计的有效性。我们的方法采用信息最大化和端到端设计原则。