Lip-to-Speech (Lip2Speech) synthesis, which predicts corresponding speech
from talking face images, has witnessed significant progress with various
models and training strategies in a series of independent studies. However,
existing studies can not achieve voice control under zero-shot condition,
because extra speaker embeddings need to be extracted from natural reference
speech and are unavailable when only the silent video of an unseen speaker is
given. In this paper, we propose a zero-shot personalized Lip2Speech synthesis
method, in which face images control speaker identities. A variational
autoencoder is adopted to disentangle the speaker identity and linguistic
content representations, which enables speaker embeddings to control the voice
characteristics of synthetic speech for unseen speakers. Furthermore, we
propose associated cross-modal representation learning to promote the ability
of face-based speaker embeddings (FSE) on voice control. Extensive experiments
verify the effectiveness of the proposed method whose synthetic utterances are
more natural and matching with the personality of input video than the compared
methods. To our best knowledge, this paper makes the first attempt on zero-shot
personalized Lip2Speech synthesis with a face image rather than reference audio
to control voice characteristics.

本文提出了一种以面部图像控制语音的零样本个性化 Lip2Speech 合成方法，并采用变分自编码器对讲话人身份和语言内容进行解藕，由此实现了对未知说话人的声音特征进行控制。此外，我们还探索了跨模态表示学习以提高面部说话人嵌入（FSE）的语音控制能力。通过大量实验证明了该方法的有效性，其合成话语比其他方法更加自然且与输入视频的个性更匹配。据我们所知，这是第一篇使用面部图像而不是参考音频来控制声音特征的零样本个性化 Lip2Speech 合成方法。

基于面部图像的语音控制的零样本个性化唇语合成

Zero-shot personalized lip-to-speech synthesis with face image based  voice control

Keyword spotting (KWS) based on deep neural networks (DNNs) has achieved
massive success in voice control scenarios. However, training of such DNN-based
KWS systems often requires significant data and hardware resources.
Manufacturers often entrust this process to a third-party platform. This makes
the training process uncontrollable, where attackers can implant backdoors in
the model by manipulating third-party training data. An effective backdoor
attack can force the model to make specified judgments under certain
conditions, i.e., triggers. In this paper, we design a backdoor attack scheme
based on Voiceprint Selection and Voice Conversion, abbreviated as VSVC.
Experimental results demonstrated that VSVC is feasible to achieve an average
attack success rate close to 97% in four victim models when poisoning less than
1% of the training data.

使用 Voiceprint Selection 和 Voice Conversion 的后门攻击方案可以成功地攻击基于深度神经网络的关键词识别系统，从而引起误判。

基于声纹选择和语音转换的关键词检测后门攻击

VSVC: Backdoor attack against Keyword Spotting based on Voiceprint Selection and Voice Conversion

Adversarial attacks against commercial black-box speech platforms, including
cloud speech APIs and voice control devices, have received little attention
until recent years. The current "black-box" attacks all heavily rely on the
knowledge of prediction/confidence scores to craft effective adversarial
examples, which can be intuitively defended by service providers without
returning these messages. In this paper, we propose two novel adversarial
attacks in more practical and rigorous scenarios. For commercial cloud speech
APIs, we propose Occam, a decision-only black-box adversarial attack, where
only final decisions are available to the adversary. In Occam, we formulate the
decision-only AE generation as a discontinuous large-scale global optimization
problem, and solve it by adaptively decomposing this complicated problem into a
set of sub-problems and cooperatively optimizing each one. Our Occam is a
one-size-fits-all approach, which achieves 100% success rates of attacks with
an average SNR of 14.23dB, on a wide range of popular speech and speaker
recognition APIs, including Google, Alibaba, Microsoft, Tencent, iFlytek, and
Jingdong, outperforming the state-of-the-art black-box attacks. For commercial
voice control devices, we propose NI-Occam, the first non-interactive physical
adversarial attack, where the adversary does not need to query the oracle and
has no access to its internal information and training data. We combine
adversarial attacks with model inversion attacks, and thus generate the
physically-effective audio AEs with high transferability without any
interaction with target devices. Our experimental results show that NI-Occam
can successfully fool Apple Siri, Microsoft Cortana, Google Assistant, iFlytek
and Amazon Echo with an average SRoA of 52% and SNR of 9.65dB, shedding light
on non-interactive physical attacks against voice control devices.

本文提出了两种新的针对商用云语音 API 和语音控制设备的对抗攻击方法：Occam 和 NI-Occam。在 Occam 中，我们将决策生成视为一个复杂的离散优化问题，通过适应性分解来协作地优化，并在多种语音和说话人识别 API 上取得了 100% 的攻击成功率。在 NI-Occam 中，我们将对抗攻击与模型反演攻击相结合，成功欺骗了多种语音控制设备。