CRATE, a white-box transformer architecture designed to learn compressed and
sparse representations, offers an intriguing alternative to standard vision
transformers (ViTs) due to its inherent mathematical interpretability. Despite
extensive investigations into the scaling behaviors of language and vision
transformers, the scalability of CRATE remains an open question which this
paper aims to address. Specifically, we propose CRATE-$\alpha$, featuring
strategic yet minimal modifications to the sparse coding block in the CRATE
architecture design, and a light training recipe designed to improve the
scalability of CRATE. Through extensive experiments, we demonstrate that
CRATE-$\alpha$ can effectively scale with larger model sizes and datasets. For
example, our CRATE-$\alpha$-B substantially outperforms the prior best CRATE-B
model accuracy on ImageNet classification by 3.7%, achieving an accuracy of
83.2%. Meanwhile, when scaling further, our CRATE-$\alpha$-L obtains an
ImageNet classification accuracy of 85.1%. More notably, these model
performance improvements are achieved while preserving, and potentially even
enhancing the interpretability of learned CRATE models, as we demonstrate
through showing that the learned token representations of increasingly larger
trained CRATE-$\alpha$ models yield increasingly higher-quality unsupervised
object segmentation of images. The project page is
this https URL

CRATE-$\alpha$ 是一种基于稀疏推理的白盒变换器架构，通过对 CRATE 结构进行小幅改进和轻量级训练，能够实现较大模型规模和数据集的可扩展性，同时保持模型的可解释性和提高模型性能，例如在 ImageNet 分类上的准确率达到 83.2% 或 85.1%。

为视觉领域扩展白盒变压器

Scaling White-Box Transformers for Vision

Transformer-like models for vision tasks have recently proven effective for a
wide range of downstream applications such as segmentation and detection.
Previous works have shown that segmentation properties emerge in vision
transformers (ViTs) trained using self-supervised methods such as DINO, but not
in those trained on supervised classification tasks. In this study, we probe
whether segmentation emerges in transformer-based models solely as a result of
intricate self-supervised learning mechanisms, or if the same emergence can be
achieved under much broader conditions through proper design of the model
architecture. Through extensive experimental results, we demonstrate that when
employing a white-box transformer-like architecture known as CRATE, whose
design explicitly models and pursues low-dimensional structures in the data
distribution, segmentation properties, at both the whole and parts levels,
already emerge with a minimalistic supervised training recipe. Layer-wise
finer-grained analysis reveals that the emergent properties strongly
corroborate the designed mathematical functions of the white-box network. Our
results suggest a path to design white-box foundation models that are
simultaneously highly performant and mathematically fully interpretable. Code
is at https://github.com/Ma-Lab-Berkeley/CRATE.

使用 CRATE 模型，在最小化的有监督训练过程中，展现了分割性质的出现，表明白盒模型的设计路径可以同时实现高性能和数学完全可解释性。