Modern language models (LMs) can learn to perform new tasks in different
ways: in instruction following, the target task is described explicitly in
natural language; in few-shot prompting, the task is specified implicitly with
a small number of examples; in instruction inference, LMs are presented with
in-context examples and are then prompted to generate a natural language task
description before making predictions. Each of these procedures may be thought
of as invoking a different form of reasoning: instruction following involves
deductive reasoning, few-shot prompting involves inductive reasoning, and
instruction inference involves abductive reasoning. How do these different
capabilities relate? Across four LMs (from the gpt and llama families) and two
learning problems (involving arithmetic functions and machine translation) we
find a strong dissociation between the different types of reasoning: LMs can
sometimes learn effectively from few-shot prompts even when they are unable to
explain their own prediction rules; conversely, they sometimes infer useful
task descriptions while completely failing to learn from human-generated
descriptions of the same task. Our results highlight the non-systematic nature
of reasoning even in some of today's largest LMs, and underscore the fact that
very different learning mechanisms may be invoked by seemingly similar
prompting procedures.

通过研究不同类型的推理方式，以及对语言模型进行的指令跟随、少样本提示和指令推断实验，我们发现即使在一些最大的语言模型中，推理的方式仍然是非系统性的，不同的学习机制可能被看似相似的提示程序调用。