While instruction-tuned models have shown remarkable success in various
natural language processing tasks, accurately evaluating their ability to
follow instructions remains challenging. Existing benchmarks primarily focus on
common instructions that align well with what the model learned during
training. However, proficiency in responding to these instructions does not
necessarily imply strong ability in instruction following. In this paper, we
propose a novel instruction-following evaluation protocol called verbalizer
manipulation. It instructs the model to verbalize the task label with words
aligning with model priors to different extents, adopting verbalizers from
highly aligned (e.g., outputting ``postive'' for positive sentiment), to
minimally aligned (e.g., outputting ``negative'' for positive sentiment).
Verbalizer manipulation can be seamlessly integrated with any classification
benchmark to examine the model's reliance on priors and its ability to override
them to accurately follow the instructions. We conduct a comprehensive
evaluation of four major model families across nine datasets, employing twelve
sets of verbalizers for each of them. We observe that the instruction-following
abilities of models, across different families and scales, are significantly
distinguished by their performance on less natural verbalizers. Even the
strongest GPT-4 model struggles to perform better than random guessing on the
most challenging verbalizer, emphasizing the need for continued advancements to
improve their instruction-following abilities.

我们提出了一种名为 verbalizer manipulation 的新型指令跟踪评估协议，该协议通过使用高度匹配到轻微匹配的语言转化器，结合任何分类基准来检查模型对先验知识的依赖程度以及其是否能准确遵循指令。我们对四个重要的模型系列进行了全面评估，使用了每个系列的十二组语言转化器，发现模型在不同系列和规模上的指令遵循能力在较不自然的语言转化器上有明显差异。即使是最强大的 GPT-4 模型在最具挑战性的语言转化器上也难以比随机猜测更好，强调了需要持续进步来改善其指令遵循能力。