Multiple-choice question answering (MCQA) is a key competence of performant transformer language models that is tested by mainstream benchmarks. However, recent evidence shows that models can have quite a range of performance, particularly when the task format is diversified slightly (such as by shuffling answer choice order). In this work we ask: how do successful models perform formatted MCQA? We employ vocabulary projection and activation patching methods to localize key hidden states that encode relevant information for predicting the correct answer. We find that prediction of a specific answer symbol is causally attributed to a single middle layer, and specifically its multi-head self-attention mechanism. We show that subsequent layers increase the probability of the predicted answer symbol in vocabulary space, and that this probability increase is associated with a sparse set of attention heads with unique roles. We additionally uncover differences in how different models adjust to alternative symbols. Finally, we demonstrate that a synthetic task can disentangle sources of model error to pinpoint when a model has learned formatted MCQA, and show that an inability to separate answer symbol tokens in vocabulary space is a property of models unable to perform formatted MCQA tasks.

本研究解决了变换器语言模型在多项选择问答（MCQA）中的表现差异问题，尤其是在任务格式略有变化的情况下。通过词汇投影和激活修补方法，研究揭示了模型在预测正确答案时关键隐藏状态的作用，并指出特定中间层及其多头自注意机制对答复的预测具有因果关系，揭示了不同模型在处理替代符号时的调整差异。该工作对理解和改善MCQA能力具有重要影响。

回答、组装、精通：理解变换器如何回答多项选择问题