Large language models such as GPT-3 (Brown et al., 2020) can perform
arbitrary tasks without undergoing fine-tuning after being prompted with only a
few labeled examples. An arbitrary task can be reformulated as a natural
language prompt, and a language model can be asked to generate the completion,
indirectly performing the task in a paradigm known as prompt-based learning. To
date, emergent prompt-based learning capabilities have mainly been demonstrated
for unidirectional language models. However, bidirectional language models
pre-trained on denoising objectives such as masked language modeling produce
stronger learned representations for transfer learning. This motivates the
possibility of prompting bidirectional models, but their pre-training
objectives have made them largely incompatible with the existing prompting
paradigm. We present SAP (Sequential Autoregressive Prompting), a technique
that enables the prompting of bidirectional models. Utilizing the machine
translation task as a case study, we prompt the bidirectional mT5 model (Xue et
al., 2021) with SAP and demonstrate its few-shot and zero-shot translations
outperform the few-shot translations of unidirectional models like GPT-3 and
XGLM (Lin et al., 2021), despite mT5's approximately 50% fewer parameters. We
further show SAP is effective on question answering and summarization. For the
first time, our results demonstrate prompt-based learning is an emergent
property of a broader class of language models, rather than only unidirectional
models.

本研究提出了一种名为 SAP 的技术，旨在使双向语言模型具有提示功能。利用机器翻译任务作为案例研究，我们使用 SAP 提示双向 mT5 模型，并证明其少量示例和零量示例的翻译性能优于 unidirectional 模型，同时展示了 SAP 在问答和摘要生成中的有效性，证明提示学习是更广泛的语言模型类别的新兴属性而不仅仅是单向模型的特征。

双向语言模型也是小样本学习器

Bidirectional Language Models Are Also Few-shot Learners

Generative patent language models can assist humans to write patent text more
effectively. The question is how to measure effectiveness from a human-centric
perspective and how to improve effectiveness. In this manuscript, a simplified
design of the autocomplete function is proposed to increase effectiveness by
more than 10%. With the new design, the effectiveness of autocomplete can reach
more than 60%, which means that more than 60% of keystrokes can be saved by
autocomplete. Since writing patent text does not necessarily start from the
beginning to the end, a question is whether the generative model can assist a
user no matter where to start writing. To answer the question, the generative
models in this manuscript are pre-trained with training data in both
directions. The generative models become bidirectional. Since text generation
is bidirectional, the calculation of autocomplete effectiveness can be
bidirectional and starts from anywhere in the text. After thorough experiments,
a key finding is that the autocomplete effectiveness of a model for the same
text remains similar no matter where the calculation starts. The finding
indicates that such bidirectional models can assist a user at a similar level,
no matter where the user starts to write.

提出了一种简化的自动完成功能设计，通过该设计可以将自动完成效率提高 10% 以上，使自动完成效果超过 60%；通过双向训练与计算，在文本的任何位置都可进行双向自动完成，实验证明，同一文本的双向自动完成效果相似，可以提高人类编写专利文本的效率。

双向生成专利语言模型的有效性

The Effectiveness of Bidirectional Generative Patent Language Models

We develop the first approximate inference algorithm for 1-Best (and M-Best)
decoding in bidirectional neural sequence models by extending Beam Search (BS)
to reason about both forward and backward time dependencies. Beam Search (BS)
is a widely used approximate inference algorithm for decoding sequences from
unidirectional neural sequence models. Interestingly, approximate inference in
bidirectional models remains an open problem, despite their significant
advantage in modeling information from both the past and future. To enable the
use of bidirectional models, we present Bidirectional Beam Search (BiBS), an
efficient algorithm for approximate bidirectional inference.To evaluate our
method and as an interesting problem in its own right, we introduce a novel
Fill-in-the-Blank Image Captioning task which requires reasoning about both
past and future sentence structure to reconstruct sensible image descriptions.
We use this task as well as the Visual Madlibs dataset to demonstrate the
effectiveness of our approach, consistently outperforming all baseline methods.

开发 Bidirectional Beam Search（BiBS）算法来逼近推理双向神经序列模型中的 1-Best（和 M-Best）解码，并在可视化任务中验证该方法的高效性和效果。