We present a method for large-mask pluralistic image inpainting based on the
generative framework of discrete latent codes. Our method learns latent priors,
discretized as tokens, by only performing computations at the visible locations
of the image. This is realized by a restrictive partial encoder that predicts
the token label for each visible block, a bidirectional transformer that infers
the missing labels by only looking at these tokens, and a dedicated synthesis
network that couples the tokens with the partial image priors to generate
coherent and pluralistic complete image even under extreme mask settings.
Experiments on public benchmarks validate our design choices as the proposed
method outperforms strong baselines in both visual quality and diversity
metrics.

大掩蔽众生成图像修复方法，基于离散潜变量的生成框架，通过仅在图像可见位置执行计算的方式来学习潜在先验，利用限制性部分编码器预测每个可见块的令牌标签，双向转换器通过仅查看这些令牌推断缺失的标签，专用合成网络将令牌与部分图像先验相耦合，即使在极端掩蔽设置下也能生成连贯且多样的完整图像，公共基准测试表明所提方法在视觉质量和多样性指标上优于强基线模型。

不要深入黑暗：众多图像修复的潜在代码

Don't Look into the Dark: Latent Codes for Pluralistic Image Inpainting

Synthesizing realistic co-speech gestures is an important and yet unsolved
problem for creating believable motions that can drive a humanoid robot to
interact and communicate with human users. Such capability will improve the
impressions of the robots by human users and will find applications in
education, training, and medical services. One challenge in learning the
co-speech gesture model is that there may be multiple viable gesture motions
for the same speech utterance. The deterministic regression methods can not
resolve the conflicting samples and may produce over-smoothed or damped
motions. We proposed a two-stage model to address this uncertainty issue in
gesture synthesis by modeling the gesture segments as discrete latent codes.
Our method utilizes RQ-VAE in the first stage to learn a discrete codebook
consisting of gesture tokens from training data. In the second stage, a
two-level autoregressive transformer model is used to learn the prior
distribution of residual codes conditioned on input speech context. Since the
inference is formulated as token sampling, multiple gesture sequences could be
generated given the same speech input using top-k sampling. The quantitative
results and the user study showed the proposed method outperforms the previous
methods and is able to generate realistic and diverse gesture motions.

针对语音对应手势模型中存在的多种可能性及冲突问题，本研究提出一种使用离散潜在编码的两阶段模型引入不确定性来解决这一问题，并使用自回归变换器模型学习手势生成的先验分布，经用户考核后证明本方法生成的手势具有更加逼真和多样性。

使用离散手势令牌学习来进行共同语音手势合成

Co-Speech Gesture Synthesis using Discrete Gesture Token Learning

In many sequence learning tasks, such as program synthesis and document
summarization, a key problem is searching over a large space of possible output
sequences. We propose to learn representations of the outputs that are
specifically meant for search: rich enough to specify the desired output but
compact enough to make search more efficient. Discrete latent codes are
appealing for this purpose, as they naturally allow sophisticated combinatorial
search strategies. The latent codes are learned using a self-supervised
learning principle, in which first a discrete autoencoder is trained on the
output sequences, and then the resulting latent codes are used as intermediate
targets for the end-to-end sequence prediction task. Based on these insights,
we introduce the \emph{Latent Programmer}, a program synthesis method that
first predicts a discrete latent code from input/output examples, and then
generates the program in the target language. We evaluate the Latent Programmer
on two domains: synthesis of string transformation programs, and generation of
programs from natural language descriptions. We demonstrate that the discrete
latent representation significantly improves synthesis accuracy.

该论文提出了一种利用具有特定搜索目的的输出表示形式来解决大规模空间搜索问题的方法，通过先使用自监督训练离散自编码器，再将结果离散潜在编码用作建立端到端序列预测任务的中间目标来学习离散潜在编码，基于这些研究成果提出了一种名为 'Latent Programmer' 的程序合成方法，实验证明使用离散潜在表示显著提高了合成精度。