Although large language models (LLMs) exhibit remarkable capacity to leverage
in-context demonstrations, it is still unclear to what extent they can learn
new concepts or facts from ground-truth labels. To address this question, we
examine the capacity of instruction-tuned LLMs to follow in-context concept
guidelines for sentence labeling tasks. We design guidelines that present
different types of factual and counterfactual concept definitions, which are
used as prompts for zero-shot sentence classification tasks. Our results show
that although concept definitions consistently help in task performance, only
the larger models (with 70B parameters or more) have limited ability to work
under counterfactual contexts. Importantly, only proprietary models such as
GPT-3.5 and GPT-4 can recognize nonsensical guidelines, which we hypothesize is
due to more sophisticated alignment methods. Finally, we find that
Falcon-180B-chat is outperformed by Llama-2-70B-chat is most cases, which
indicates that careful fine-tuning is more effective than increasing model
scale. Altogether, our simple evaluation method reveals significant gaps in
concept understanding between the most capable open-source language models and
the leading proprietary APIs.

大语言模型通过指导文本对句子进行标记涉及了利用上下文示例的广泛语言模型，根据我们的简单评估方法，我们发现开源语言模型与领先的专有 API 之间在概念理解方面存在显著差距。