Formula-driven supervised learning (FDSL) has been shown to be an effective
method for pre-training vision transformers, where ExFractalDB-21k was shown to
exceed the pre-training effect of ImageNet-21k. These studies also indicate
that contours mattered more than textures when pre-training vision
transformers. However, the lack of a systematic investigation as to why these
contour-oriented synthetic datasets can achieve the same accuracy as real
datasets leaves much room for skepticism. In the present work, we develop a
novel methodology based on circular harmonics for systematically investigating
the design space of contour-oriented synthetic datasets. This allows us to
efficiently search the optimal range of FDSL parameters and maximize the
variety of synthetic images in the dataset, which we found to be a critical
factor. When the resulting new dataset VisualAtom-21k is used for pre-training
ViT-Base, the top-1 accuracy reached 83.7% when fine-tuning on ImageNet-1k.
This is close to the top-1 accuracy (84.2%) achieved by JFT-300M pre-training,
while the number of images is 1/14. Unlike JFT-300M which is a static dataset,
the quality of synthetic datasets will continue to improve, and the current
work is a testament to this possibility. FDSL is also free of the common issues
associated with real images, e.g. privacy/copyright issues, labeling
costs/errors, and ethical biases.

使用循环谐波的新方法，探讨了基于轮廓的合成数据集的设计空间，发现增加数据集中的图像数量和种类是关键因素，在新数据集 VisualAtom-21k 上进行预训练后，fine-tuning 在 ImageNet-1k 上的 top-1 准确率达到了 83.7％，与 JFT-300M 预训练相比，图像数量缩小到 1/14，解决了真实图像所带来的问题。