Cued Speech (CS) is an advanced visual phonetic encoding system that
integrates lip reading with hand codings, enabling people with hearing
impairments to communicate efficiently. CS video generation aims to produce
specific lip and gesture movements of CS from audio or text inputs. The main
challenge is that given limited CS data, we strive to simultaneously generate
fine-grained hand and finger movements, as well as lip movements, meanwhile the
two kinds of movements need to be asynchronously aligned. Existing CS
generation methods are fragile and prone to poor performance due to
template-based statistical models and careful hand-crafted pre-processing to
fit the models. Therefore, we propose a novel Gloss-prompted Diffusion-based CS
Gesture generation framework (called GlossDiff). Specifically, to integrate
additional linguistic rules knowledge into the model. we first introduce a
bridging instruction called \textbf{Gloss}, which is an automatically generated
descriptive text to establish a direct and more delicate semantic connection
between spoken language and CS gestures. Moreover, we first suggest rhythm is
an important paralinguistic feature for CS to improve the communication
efficacy. Therefore, we propose a novel Audio-driven Rhythmic Module (ARM) to
learn rhythm that matches audio speech. Moreover, in this work, we design,
record, and publish the first Chinese CS dataset with four CS cuers. Extensive
experiments demonstrate that our method quantitatively and qualitatively
outperforms current state-of-the-art (SOTA) methods. We release the code and
data at this https URL

提出了一种基于 GlossDiff 框架的 Cued Speech（CS）手势生成方法，该方法通过引入 Gloss 和 Audio-driven Rhythmic Module（ARM）等新技术，实现了对 CS 手势的精细生成和节奏匹配，同时发布了第一个中文 CS 数据集。实验证明该方法在定量和定性上超越了当前的最新技术。