Formula-driven supervised learning (FDSL) is a pre-training method that
relies on synthetic images generated from mathematical formulae such as
fractals. Prior work on FDSL has shown that pre-training vision transformers on
such synthetic datasets can yield competitive accuracy on a wide range of
downstream tasks. These synthetic images are categorized according to the
parameters in the mathematical formula that generate them. In the present work,
we hypothesize that the process for generating different instances for the same
category in FDSL, can be viewed as a form of data augmentation. We validate
this hypothesis by replacing the instances with data augmentation, which means
we only need a single image per category. Our experiments shows that this
one-instance fractal database (OFDB) performs better than the original dataset
where instances were explicitly generated. We further scale up OFDB to 21,000
categories and show that it matches, or even surpasses, the model pre-trained
on ImageNet-21k in ImageNet-1k fine-tuning. The number of images in OFDB is
21k, whereas ImageNet-21k has 14M. This opens new possibilities for
pre-training vision transformers with much smaller datasets.

基于数学公式生成的合成图像的预训练方法（FDSL）通过数据增强替代了显式生成实例，证实其性能优于原始数据集，从而打开了使用更小数据集进行预训练视觉转换的新可能性。

用非常有限的合成图像预训练视觉 Transformer

Pre-training Vision Transformers with Very Limited Synthesized Images

Is it possible to use convolutional neural networks pre-trained without any
natural images to assist natural image understanding? The paper proposes a
novel concept, Formula-driven Supervised Learning. We automatically generate
image patterns and their category labels by assigning fractals, which are based
on a natural law existing in the background knowledge of the real world.
Theoretically, the use of automatically generated images instead of natural
images in the pre-training phase allows us to generate an infinite scale
dataset of labeled images. Although the models pre-trained with the proposed
Fractal DataBase (FractalDB), a database without natural images, does not
necessarily outperform models pre-trained with human annotated datasets at all
settings, we are able to partially surpass the accuracy of ImageNet/Places
pre-trained models. The image representation with the proposed FractalDB
captures a unique feature in the visualization of convolutional layers and
attentions.

该论文提出了一种新颖的概念 —— 公式驱动的监督学习，利用分形形象自动生成命名类别标签，从而实现图像预训练，尽管所提出的 FractalDB 预训练模型在所有设置中并未完全胜过人类注释的数据集，但在部分情形下其预测准确率优于 ImageNet/Places 预训练模型，并且所提出的 FractalDB 可以捕捉卷积层和注意力可视化中的独特特征。