Transfer learning improves the performance of deep learning models by initializing them with parameters pre-trained on larger datasets. Intuitively, transfer learning is more effective when pre-training is on the in-domain datasets. A recent study by NASA has demonstrated that the microstructure segmentation with encoder-decoder algorithms benefits more from CNN encoders pre-trained on microscopy images than from those pre-trained on natural images. However, CNN models only capture the local spatial relations in images. In recent years, attention networks such as Transformers are increasingly used in image analysis to capture the long-range relations between pixels. In this study, we compare the segmentation performance of Transformer and CNN models pre-trained on microscopy images with those pre-trained on natural images. Our result partially confirms the NASA study that the segmentation performance of out-of-distribution images (taken under different imaging and sample conditions) is significantly improved when pre-training on microscopy images. However, the performance gain for one-shot and few-shot learning is more modest with Transformers. We also find that for image segmentation, the combination of pre-trained Transformers and CNN encoders are consistently better than pre-trained CNN encoders alone. Our dataset (of about 50,000 images) combines the public portion of the NASA dataset with additional images we collected. Even with much less training data, our pre-trained models have significantly better performance for image segmentation. This result suggests that Transformers and CNN complement each other and when pre-trained on microscopy images, they are more beneficial to the downstream tasks.

转移学习通过使用在更大数据集上预先训练的参数来提高深度学习模型的性能。最近NASA的一项研究表明，使用在显微图像上预先训练的CNN编码器比使用在自然图像上预先训练的CNN编码器更有益于显微结构分割。然而，仅使用CNN模型仅能捕捉图像中的局部空间关系。本研究对比了在显微图像和自然图像上预先训练的Transformers和CNN模型的分割性能，结果部分确认了NASA研究的结论。我们还发现，在图像分割中，预先训练的Transformers和CNN编码器的组合一直优于仅预先训练的CNN编码器。这些结果表明，Transformers和CNN相互补充，并且当在显微图像上预先训练时，它们对下游任务更有益。

CS-UNet模型的显微结构分割的迁移学习：一种基于Transformer和CNN编码器的混合算法