This paper describes MAIA, a Multimodal Automated Interpretability Agent.
MAIA is a system that uses neural models to automate neural model understanding
tasks like feature interpretation and failure mode discovery. It equips a
pre-trained vision-language model with a set of tools that support iterative
experimentation on subcomponents of other models to explain their behavior.
These include tools commonly used by human interpretability researchers: for
synthesizing and editing inputs, computing maximally activating exemplars from
real-world datasets, and summarizing and describing experimental results.
Interpretability experiments proposed by MAIA compose these tools to describe
and explain system behavior. We evaluate applications of MAIA to computer
vision models. We first characterize MAIA's ability to describe (neuron-level)
features in learned representations of images. Across several trained models
and a novel dataset of synthetic vision neurons with paired ground-truth
descriptions, MAIA produces descriptions comparable to those generated by
expert human experimenters. We then show that MAIA can aid in two additional
interpretability tasks: reducing sensitivity to spurious features, and
automatically identifying inputs likely to be mis-classified.

这篇论文介绍了 MAIA，一种多模态自动解释性代理。MAIA 是一个使用神经模型自动化神经模型理解任务的系统，如特征解释和故障模式发现。它通过提供一系列工具来对其他模型的子组件进行迭代实验，从而对其行为进行解释。这些工具包括人工解释性研究人员常用的工具：用于合成和编辑输入、计算最大激活样本、以及总结和描述实验结果。MAIA 提出的解释性实验将这些工具组合起来描述和解释系统行为。我们评估了 MAIA 在计算机视觉模型中的应用。首先，我们描述了 MAIA 在学习到的图像表示的特征（神经元级别）中的描述能力。在多个经过训练的模型和一个包含真实描述的人工生成视觉神经元数据集中，MAIA 生成了与由专家人工实验者生成的描述相当的结果。然后，我们展示了 MAIA 在两个附加的可解释性任务中的应用：降低对虚假特征的敏感性和自动识别可能被错误分类的输入。

多模态自动可解释性代理

A Multimodal Automated Interpretability Agent

As artificial intelligence becomes increasingly intelligent---in some cases,
achieving superhuman performance---there is growing potential for humans to
learn from and collaborate with algorithms. However, the ways in which AI
systems approach problems are often different from the ways people do, and thus
may be uninterpretable and hard to learn from. A crucial step in bridging this
gap between human and artificial intelligence is modeling the granular actions
that constitute human behavior, rather than simply matching aggregate human
performance.
We pursue this goal in a model system with a long history in artificial
intelligence: chess. The aggregate performance of a chess player unfolds as
they make decisions over the course of a game. The hundreds of millions of
games played online by players at every skill level form a rich source of data
in which these decisions, and their exact context, are recorded in minute
detail. Applying existing chess engines to this data, including an open-source
implementation of AlphaZero, we find that they do not predict human moves well.
We develop and introduce Maia, a customized version of Alpha-Zero trained on
human chess games, that predicts human moves at a much higher accuracy than
existing engines, and can achieve maximum accuracy when predicting decisions
made by players at a specific skill level in a tuneable way. For a dual task of
predicting whether a human will make a large mistake on the next move, we
develop a deep neural network that significantly outperforms competitive
baselines. Taken together, our results suggest that there is substantial
promise in designing artificial intelligence systems with human collaboration
in mind by first accurately modeling granular human decision-making.

研究人员基于人类下棋决策的详细数据，通过 AlphaZero 构建了一种定制化的人工智能引擎 Maia，能够更准确地预测人类棋手的下棋。这项研究表明，重点是建立精确的人类决策模型，安装有人类协作的人工智能系统具有很大的潜力。