While vision models are highly capable, their internal mechanisms remain poorly understood -- a challenge which sparse autoencoders (SAEs) have helped address in language, but which remains underexplored in vision. We address this gap by training SAEs on CLIP's vision transformer and uncover key differences between vision and language processing, including distinct sparsity patterns for SAEs trained across layers and token types. We then provide the first systematic analysis on the steerability of CLIP's vision transformer by introducing metrics to quantify how precisely SAE features can be steered to affect the model's output. We find that 10-15\% of neurons and features are steerable, with SAEs providing thousands more steerable features than the base model. Through targeted suppression of SAE features, we then demonstrate improved performance on three vision disentanglement tasks (CelebA, Waterbirds, and typographic attacks), finding optimal disentanglement in middle model layers, and achieving state-of-the-art performance on defense against typographic attacks.

本研究解决了视觉模型内部机制理解不足的问题，通过在CLIP的视觉变换器上训练稀疏自编码器（SAEs），揭示了视觉与语言处理之间的关键差异，尤其是各层和token类型的稀疏性模式。研究还系统分析了CLIP视觉变换器的可引导性，发现10-15%的神经元和特征可被引导，从而在三项视觉解耦任务中（CelebA, Waterbirds和排版攻击）提升了性能，实现了领先的防御能力。

利用稀疏自编码器引导CLIP的视觉变换器