In this work, we address the problem of generating speech from silent lip
videos for any speaker in the wild. In stark contrast to previous works, our
method (i) is not restricted to a fixed number of speakers, (ii) does not
explicitly impose constraints on the domain or the vocabulary and (iii) deals
with videos that are recorded in the wild as opposed to within laboratory
settings. The task presents a host of challenges, with the key one being that
many features of the desired target speech, like voice, pitch and linguistic
content, cannot be entirely inferred from the silent face video. In order to
handle these stochastic variations, we propose a new VAE-GAN architecture that
learns to associate the lip and speech sequences amidst the variations. With
the help of multiple powerful discriminators that guide the training process,
our generator learns to synthesize speech sequences in any voice for the lip
movements of any person. Extensive experiments on multiple datasets show that
we outperform all baselines by a large margin. Further, our network can be
fine-tuned on videos of specific identities to achieve a performance comparable
to single-speaker models that are trained on $4\times$ more data. We conduct
numerous ablation studies to analyze the effect of different modules of our
architecture. We also provide a demo video that demonstrates several
qualitative results along with the code and trained models on our website:
http://cvit.iiit.ac.in/research/projects/cvit-projects/lip-to-speech-synthesis}

采用新型的 VAE-GAN 架构生成多说话者在野外拍摄的沉默唇视频中的语音，比其他基线模型表现更好，能够实现针对特定身份的微调和生成不同语音的语音序列。

野外任意说话人唇语合成

Lip-to-Speech Synthesis for Arbitrary Speakers in the Wild

We propose a technique for multi-task learning from demonstration that trains
the controller of a low-cost robotic arm to accomplish several complex picking
and placing tasks, as well as non-prehensile manipulation. The controller is a
recurrent neural network using raw images as input and generating robot arm
trajectories, with the parameters shared across the tasks. The controller also
combines VAE-GAN-based reconstruction with autoregressive multimodal action
prediction. Our results demonstrate that it is possible to learn complex
manipulation tasks, such as picking up a towel, wiping an object, and
depositing the towel to its previous position, entirely from raw images with
direct behavior cloning. We show that weight sharing and reconstruction-based
regularization substantially improve generalization and robustness, and
training on multiple tasks simultaneously increases the success rate on all
tasks.

以多任务学习为基础，提出一种通过演示学习从而训练低成本机械臂控制器，以便完成数个拾取放置任务及非预抓取式操控操作的技术，使用原始图像作为输入并生成机器人臂路径的基于循环神经网络的控制器，参数在任务之间共享；同时结合基于 VAE-GAN 重建以及自回归多模态行为预测的控制器模型。结果表明，可以通过行为克隆直接从原始图像中学习复杂的操纵任务，例如拾起毛巾、擦拭物体并将毛巾放回原位等，权重共享和基于重构的正则化大大提高了泛化性和鲁棒性，同时同时训练多个任务能够增加所有任务的成功率。

基于端到端演示学习的基于视觉的廉价机器人多任务操作

Vision-Based Multi-Task Manipulation for Inexpensive Robots Using  End-To-End Learning from Demonstration

We explore methods of producing adversarial examples on deep generative
models such as the variational autoencoder (VAE) and the VAE-GAN. Deep learning
architectures are known to be vulnerable to adversarial examples, but previous
work has focused on the application of adversarial examples to classification
tasks. Deep generative models have recently become popular due to their ability
to model input data distributions and generate realistic examples from those
distributions. We present three classes of attacks on the VAE and VAE-GAN
architectures and demonstrate them against networks trained on MNIST, SVHN and
CelebA. Our first attack leverages classification-based adversaries by
attaching a classifier to the trained encoder of the target generative model,
which can then be used to indirectly manipulate the latent representation. Our
second attack directly uses the VAE loss function to generate a target
reconstruction image from the adversarial example. Our third attack moves
beyond relying on classification or the standard loss for the gradient and
directly optimizes against differences in source and target latent
representations. We also motivate why an attacker might be interested in
deploying such techniques against a target generative network.

本篇研究探索在变分自编码器（VAE）和 VAE-GAN 等深度生成模型中生成对抗性样本的方法，并提出了三种攻击方式来测试针对 MNIST、SVHN 和 CelebA 数据集训练的网络，分别利用基于分类的对手，直接利用 VAE 的损失函数生成重构图像，以及直接优化源和目标潜在表示之间的差异。