Some neurons in deep networks specialize in recognizing highly specific
perceptual, structural, or semantic features of inputs. In computer vision,
techniques exist for identifying neurons that respond to individual concept
categories like colors, textures, and object classes. But these techniques are
limited in scope, labeling only a small subset of neurons and behaviors in any
network. Is a richer characterization of neuron-level computation possible? We
introduce a procedure (called MILAN, for mutual-information-guided linguistic
annotation of neurons) that automatically labels neurons with open-ended,
compositional, natural language descriptions. Given a neuron, MILAN generates a
description by searching for a natural language string that maximizes pointwise
mutual information with the image regions in which the neuron is active. MILAN
produces fine-grained descriptions that capture categorical, relational, and
logical structure in learned features. These descriptions obtain high agreement
with human-generated feature descriptions across a diverse set of model
architectures and tasks, and can aid in understanding and controlling learned
models. We highlight three applications of natural language neuron
descriptions. First, we use MILAN for analysis, characterizing the distribution
and importance of neurons selective for attribute, category, and relational
information in vision models. Second, we use MILAN for auditing, surfacing
neurons sensitive to human faces in datasets designed to obscure them. Finally,
we use MILAN for editing, improving robustness in an image classifier by
deleting neurons sensitive to text features spuriously correlated with class
labels.

该研究提出了一种名为 MILAN 的程序，通过相互信息引导的语言注释对神经元进行自动标记，并可以生成细粒度的描述，以促进对神经网络的理解、分析和控制，并提供了自然语言神经元描述的三个应用实例。