Language models (LMs) have proven to be powerful tools for psycholinguistic research, but most prior work has focused on purely behavioural measures (e.g., surprisal comparisons). At the same time, research in model interpretability has begun to illuminate the abstract causal mechanisms shaping LM behavior. To help bring these strands of research closer together, we introduce CausalGym. We adapt and expand the SyntaxGym suite of tasks to benchmark the ability of interpretability methods to causally affect model behaviour. To illustrate how CausalGym can be used, we study the pythia models (14M--6.9B) and assess the causal efficacy of a wide range of interpretability methods, including linear probing and distributed alignment search (DAS). We find that DAS outperforms the other methods, and so we use it to study the learning trajectory of two difficult linguistic phenomena in pythia-1b: negative polarity item licensing and filler--gap dependencies. Our analysis shows that the mechanism implementing both of these tasks is learned in discrete stages, not gradually.

语言模型对于心理语言学研究具有重要作用，该研究提出了CausalGym框架，通过评估多种解释性方法的因果有效性来研究语言模型的行为，并发现DAS方法在性能上优于其他方法。在此基础上，用pythia模型研究了负极性项许可和填充-间隙依赖这两个困难的语言现象，并分析表明这两个任务的实现机制是通过离散阶段学习而非逐渐学习。

CausalGym: 在语言任务上基准测试因果解释方法