Self-supervision has been widely explored as a means of addressing the lack
of inductive biases in vision transformer architectures, which limits
generalisation when networks are trained on small datasets. This is crucial in
the context of cortical imaging, where phenotypes are complex and
heterogeneous, but the available datasets are limited in size. This paper
builds upon recent advancements in translating vision transformers to surface
meshes and investigates the potential of Masked AutoEncoder (MAE)
self-supervision for cortical surface learning. By reconstructing surface data
from a masked version of the input, the proposed method effectively models
cortical structure to learn strong representations that translate to improved
performance in downstream tasks. We evaluate our approach on cortical phenotype
regression using the developing Human Connectome Project (dHCP) and demonstrate
that pre-training leads to a 26\% improvement in performance, with an 80\%
faster convergence, compared to models trained from scratch. Furthermore, we
establish that pre-training vision transformer models on large datasets, such
as the UK Biobank (UKB), enables the acquisition of robust representations for
finetuning in low-data scenarios. Our code and pre-trained models are publicly
available at https://github.com/metrics-lab/surface-vision-transformers.

本研究通过使用自监督学习以及具有表面网格的视觉转换器架构，构建了模型以模拟皮层结构，进而实现对皮层表面的学习。通过在较大数据集上进行预训练，可以获得强大的表示能力，用于在数据少的情景下进行微调，而在皮层表型回归任务中，预训练可以带来 26% 的性能提升和 80% 的更快收敛速度。