Large language models demonstrate a remarkable capability for learning to
solve new tasks from a few examples. The prompt template, or the way the input
examples are formatted to obtain the prompt, is an important yet often
overlooked aspect of in-context learning. In this work, we conduct a
comprehensive study of the template format's influence on the in-context
learning performance. We evaluate the impact of the prompt template across
models (from 770M to 70B parameters) and 4 standard classification datasets. We
show that a poor choice of the template can reduce the performance of the
strongest models and inference methods to a random guess level. More
importantly, the best templates do not transfer between different setups and
even between models of the same family. Our findings show that the currently
prevalent approach to evaluation, which ignores template selection, may give
misleading results due to different templates in different works. As a first
step towards mitigating this issue, we propose Template Ensembles that
aggregate model predictions across several templates. This simple test-time
augmentation boosts average performance while being robust to the choice of
random set of templates.

大语言模型在解决新任务方面表现出了非凡的能力。本研究在大语言模型的上下文学习性能中探究了输入示例的格式对模型性能的影响，并提出了使用多个模板进行模型预测的模板集成方法，以提高模型性能并减少模板选择的影响。