The wayward quality of continuous prompts stresses the importance of their
interpretability as unexpected and unpredictable behaviors appear following
training, especially in the context of large language models automating
people-sensitive tasks such as resume screening. In this paper we present a
novel method of constructing continuous prompts via discrete prompt embeddings
and evaluate improvements to continuous prompt interpretability and inference
accuracy. For a set of manually designed discrete prompts $\mathcal{D}$, which
we tokenize each into tensor form, we train a model to predict the weights such
that the linear combinations of those prompts correspond to higher performance
on natural language understanding tasks.

通过离散提示嵌入的方法，我们提出了一种构建连续提示的新方法，以提高连续提示的可解释性和推理准确性，验证了在自然语言理解任务中使用线性组合这些提示会获得更好性能的权重预测模型。

离散提示嵌入的连续提示生成

Continuous Prompt Generation from Linear Combination of Discrete Prompt  Embeddings

A prompt is a sequence of symbol or tokens, selected from a vocabulary
according to some rule, which is prepended/concatenated to a textual query. A
key problem is how to select the sequence of tokens: in this paper we formulate
it as a combinatorial optimization problem. The high dimensionality of the
token space com-pounded by the length of the prompt sequence requires a very
efficient solution. In this paper we propose a Bayesian optimization method,
executed in a continuous em-bedding of the combinatorial space. In this paper
we focus on hard prompt tuning (HPT) which directly searches for discrete
tokens to be added to the text input with-out requiring access to the large
language model (LLM) and can be used also when LLM is available only as a
black-box. This is critically important if LLMs are made available in the Model
as a Service (MaaS) manner as in GPT-4. The current manu-script is focused on
the optimization of discrete prompts for classification tasks. The discrete
prompts give rise to difficult combinatorial optimization problem which easily
become intractable given the dimension of the token space in realistic
applications. The optimization method considered in this paper is Bayesian
optimization (BO) which has become the dominant approach in black-box
optimization for its sample efficiency along with its modular structure and
versatility. In this paper we use BoTorch, a library for Bayesian optimization
research built on top of pyTorch. Albeit preliminary and obtained using a
'vanilla' version of BO, the experiments on RoB-ERTa on six benchmarks, show a
good performance across a variety of tasks and enable an analysis of the
tradeoff between size of the search space, accuracy and wall clock time.

以贝叶斯优化为基础，研究离散提示的优化方法及其应用于分类任务中的性能表现。

预训练语言模型中的提示优化的贝叶斯方法

A Bayesian approach for prompt optimization in pre-trained language  models

Discrete prompts have been used for fine-tuning Pre-trained Language Models
for diverse NLP tasks. In particular, automatic methods that generate discrete
prompts from a small set of training instances have reported superior
performance. However, a closer look at the learnt prompts reveals that they
contain noisy and counter-intuitive lexical constructs that would not be
encountered in manually-written prompts. This raises an important yet
understudied question regarding the robustness of automatically learnt discrete
prompts when used in downstream tasks. To address this question, we conduct a
systematic study of the robustness of discrete prompts by applying carefully
designed perturbations into an application using AutoPrompt and then measure
their performance in two Natural Language Inference (NLI) datasets. Our
experimental results show that although the discrete prompt-based method
remains relatively robust against perturbations to NLI inputs, they are highly
sensitive to other types of perturbations such as shuffling and deletion of
prompt tokens. Moreover, they generalize poorly across different NLI datasets.
We hope our findings will inspire future work on robust discrete prompt
learning.

本文研究了离散提示在下游自然语言推理任务中的鲁棒性，通过应用 AutoPrompt 并在两个自然语言推理数据集中测量其表现，发现尽管基于离散提示的方法对自然语言推理输入的扰动相对较为鲁棒，但对提示令牌的移动和删除等其他类型的扰动非常敏感，而且其在不同的自然语言推理数据集上的泛化能力较差。