This paper explores the in-context learning capabilities of masked language
models, challenging the common view that this ability does not 'emerge' in
them. We present an embarrassingly simple inference technique that enables
DeBERTa to operate as a generative model without any additional training. Our
findings demonstrate that DeBERTa can match and even surpass GPT-3, its
contemporary that famously introduced the paradigm of in-context learning. The
comparative analysis reveals that the masked and causal language models behave
very differently, as they clearly outperform each other on different categories
of tasks. This suggests that there is great potential for a hybrid training
approach that takes advantage of the strengths of both training objectives.

本文探讨了掩码语言模型的上下文学习能力，挑战了常见观点，即这种能力在它们中并没有 ' 出现 '。我们提出了一种令人尴尬地简单的推理技术，使得 DeBERTa 能够作为一个生成模型进行操作，无需额外训练。我们的研究结果表明，DeBERTa 能够与甚至超过 GPT-3，后者以引入上下文学习范式而闻名。比较分析表明，掩码和因果语言模型的行为非常不同，它们在不同类别的任务上明显超越对方。这表明存在一个能充分利用两个训练目标优势的混合训练方法的巨大潜力。

BERTs 是生成上下文学习者

BERTs are Generative In-Context Learners

One-shot fine-grained visual recognition often suffers from the problem of
training data scarcity for new fine-grained classes. To alleviate this problem,
an off-the-shelf image generator can be applied to synthesize additional
training images, but these synthesized images are often not helpful for
actually improving the accuracy of one-shot fine-grained recognition. This
paper proposes a meta-learning framework to combine generated images with
original images, so that the resulting ``hybrid'' training images can improve
one-shot learning. Specifically, the generic image generator is updated by a
few training instances of novel classes, and a Meta Image Reinforcing Network
(MetaIRNet) is proposed to conduct one-shot fine-grained recognition as well as
image reinforcement. The model is trained in an end-to-end manner, and our
experiments demonstrate consistent improvement over baselines on one-shot
fine-grained image classification benchmarks.

本文提出了一种元学习框架，将生成的图像与原始图像相结合，形成 “混合” 训练图像以提高一次性精细识别，在一些实验中，该模型相对基准表现出了一致的改进。

元强化的合成数据在一次性细粒度视觉识别中的应用

Meta-Reinforced Synthetic Data for One-Shot Fine-Grained Visual  Recognition

Adversarial learning of probabilistic models has recently emerged as a
promising alternative to maximum likelihood. Implicit models such as generative
adversarial networks (GAN) often generate better samples compared to explicit
models trained by maximum likelihood. Yet, GANs sidestep the characterization
of an explicit density which makes quantitative evaluations challenging. To
bridge this gap, we propose Flow-GANs, a generative adversarial network for
which we can perform exact likelihood evaluation, thus supporting both
adversarial and maximum likelihood training. When trained adversarially,
Flow-GANs generate high-quality samples but attain extremely poor
log-likelihood scores, inferior even to a mixture model memorizing the training
data; the opposite is true when trained by maximum likelihood. Results on MNIST
and CIFAR-10 demonstrate that hybrid training can attain high held-out
likelihoods while retaining visual fidelity in the generated samples.

通过混合训练方式，利用生成对抗网络以及精确似然估计评估方法，解决了最大似然训练无法评估生成质量的问题。