Current literature demonstrates that Large Language Models (LLMs) are great
few-shot learners, and prompting significantly increases their performance on a
range of downstream tasks in a few-shot learning setting. An attempt to
automate human-led prompting followed, with some progress achieved. In
particular, subsequent work demonstrates automation can outperform fine-tuning
in certain K-shot learning scenarios.
In this paper, we revisit techniques for automated prompting on six different
downstream tasks and a larger range of K-shot learning settings. We find that
automated prompting does not consistently outperform simple manual prompts. Our
work suggests that, in addition to fine-tuning, manual prompts should be used
as a baseline in this line of research.

本文研究了自动提示技术在六个不同的下游任务和更广泛的 K-shot 学习场景中的应用。我们发现，自动提示并不总是优于简单的手动提示。我们的研究表明，在这一领域的研究中，除了 fine-tuning 之外还应该使用手动提示作为基线。

重新审视自动提示：我们真的做得更好了吗？

Revisiting Automated Prompting: Are We Actually Doing Better?

Feature representations from pre-trained deep neural networks have been known
to exhibit excellent generalization and utility across a variety of related
tasks. Fine-tuning is by far the simplest and most widely used approach that
seeks to exploit and adapt these feature representations to novel tasks with
limited data. Despite the effectiveness of fine-tuning, itis often sub-optimal
and requires very careful optimization to prevent severe over-fitting to small
datasets. The problem of sub-optimality and over-fitting, is due in part to the
large number of parameters used in a typical deep convolutional neural network.
To address these problems, we propose a simple yet effective regularization
method for fine-tuning pre-trained deep networks for the task of k-shot
learning. To prevent overfitting, our key strategy is to cluster the model
parameters while ensuring intra-cluster similarity and inter-cluster diversity
of the parameters, effectively regularizing the dimensionality of the parameter
search space. In particular, we identify groups of neurons within each layer of
a deep network that shares similar activation patterns. When the network is to
be fine-tuned for a classification task using only k examples, we propagate a
single gradient to all of the neuron parameters that belong to the same group.
The grouping of neurons is non-trivial as neuron activations depend on the
distribution of the input data. To efficiently search for optimal groupings
conditioned on the input data, we propose a reinforcement learning search
strategy using recurrent networks to learn the optimal group assignments for
each network layer. Experimental results show that our method can be easily
applied to several popular convolutional neural networks and improve upon other
state-of-the-art fine-tuning based k-shot learning strategies by more than10%

为了解决小规模数据下优化问题，我们提出了一种对预训练深度神经网络进行微调的简单而有效的正则化方法，它的核心策略是将模型参数聚类，确保聚类内的相似性和聚类间的差异性，提高参数搜索空间的正规性； 具体而言，我们使用强化学习搜索策略和递归网络来学习每个网络层的最优分组，实验结果表明，相较於其他最新的基于微调的 k-shot 学习策略，我们的方法可以轻松应用于几种流行的卷积神经网络，提高了 10% 以上的性能。