We propose Masked Capsule Autoencoders (MCAE), the first Capsule Network that
utilises pretraining in a self-supervised manner. Capsule Networks have emerged
as a powerful alternative to Convolutional Neural Networks (CNNs), and have
shown favourable properties when compared to Vision Transformers (ViT), but
have struggled to effectively learn when presented with more complex data,
leading to Capsule Network models that do not scale to modern tasks. Our
proposed MCAE model alleviates this issue by reformulating the Capsule Network
to use masked image modelling as a pretraining stage before finetuning in a
supervised manner. Across several experiments and ablations studies we
demonstrate that similarly to CNNs and ViTs, Capsule Networks can also benefit
from self-supervised pretraining, paving the way for further advancements in
this neural network domain. For instance, pretraining on the Imagenette
dataset, a dataset of 10 classes of Imagenet-sized images, we achieve not only
state-of-the-art results for Capsule Networks but also a 9% improvement
compared to purely supervised training. Thus we propose that Capsule Networks
benefit from and should be trained within a masked image modelling framework,
with a novel capsule decoder, to improve a Capsule Network's performance on
realistic-sized images.

我们提出了 Masked Capsule Autoencoders（MCAE），这是第一个利用自监督方式进行预训练的 Capsule Network。通过在监督训练之前使用掩码图像建模作为预训练阶段，我们的 MCAE 模型解决了 Capsule Network 在处理更复杂数据时的学习效果不佳的问题，从而取得了一系列实验证明自监督预训练对于 Capsule Networks 同样有效，为神经网络领域的进一步发展铺平了道路。