Vision transformer (ViT) and its variants have achieved remarkable successes
in various visual tasks. The key characteristic of these ViT models is to adopt
different aggregation strategies of spatial patch information within the
artificial neural networks (ANNs). However, there is still a key lack of
unified representation of different ViT architectures for systematic
understanding and assessment of model representation performance. Moreover, how
those well-performing ViT ANNs are similar to real biological neural networks
(BNNs) is largely unexplored. To answer these fundamental questions, we, for
the first time, propose a unified and biologically-plausible relational graph
representation of ViT models. Specifically, the proposed relational graph
representation consists of two key sub-graphs: aggregation graph and affine
graph. The former one considers ViT tokens as nodes and describes their spatial
interaction, while the latter one regards network channels as nodes and
reflects the information communication between channels. Using this unified
relational graph representation, we found that: a) a sweet spot of the
aggregation graph leads to ViTs with significantly improved predictive
performance; b) the graph measures of clustering coefficient and average path
length are two effective indicators of model prediction performance, especially
when applying on the datasets with small samples; c) our findings are
consistent across various ViT architectures and multiple datasets; d) the
proposed relational graph representation of ViT has high similarity with real
BNNs derived from brain science data. Overall, our work provides a novel
unified and biologically-plausible paradigm for more interpretable and
effective representation of ViT ANNs.

本文提出了视觉变换器（ViT）的统一生物学关系图表示，其中聚合图和仿射图是关键子图，本方法能够很好地描述其空间相互作用和网络通信，并发现了聚合图的组团系数和平均路径长度是模型预测性能的有效指标且具有很高的可解释性。