Recent extensions of Cellular Automata (CA) have incorporated key ideas from
modern deep learning, dramatically extending their capabilities and catalyzing
a new family of Neural Cellular Automata (NCA) techniques. Inspired by
Transformer-based architectures, our work presents a new class of
$\textit{attention-based}$ NCAs formed using a spatially
localized$\unicode{x2014}$yet globally organized$\unicode{x2014}$self-attention
scheme. We introduce an instance of this class named $\textit{Vision
Transformer Cellular Automata}$ (ViTCA). We present quantitative and
qualitative results on denoising autoencoding across six benchmark datasets,
comparing ViTCA to a U-Net, a U-Net-based CA baseline (UNetCA), and a Vision
Transformer (ViT). When comparing across architectures configured to similar
parameter complexity, ViTCA architectures yield superior performance across all
benchmarks and for nearly every evaluation metric. We present an ablation study
on various architectural configurations of ViTCA, an analysis of its effect on
cell states, and an investigation on its inductive biases. Finally, we examine
its learned representations via linear probes on its converged cell state
hidden representations, yielding, on average, superior results when compared to
our U-Net, ViT, and UNetCA baselines.

本文介绍了一种名为 Vision Transformer Cellular Automata（ViTCA）的定义在细胞自动机框架内基于自注意力机制的人工神经网络，并比较了其在线性降噪自编码任务上与其他神经网络的性能。