We present the first systematic investigation of supervised scaling laws
outside of an ImageNet-like context - on images of galaxies. We use 840k galaxy
images and over 100M annotations by Galaxy Zoo volunteers, comparable in scale
to Imagenet-1K. We find that adding annotated galaxy images provides a power
law improvement in performance across all architectures and all tasks, while
adding trainable parameters is effective only for some (typically more
subjectively challenging) tasks. We then compare the downstream performance of
finetuned models pretrained on either ImageNet-12k alone vs. additionally
pretrained on our galaxy images. We achieve an average relative error rate
reduction of 31% across 5 downstream tasks of scientific interest. Our
finetuned models are more label-efficient and, unlike their
ImageNet-12k-pretrained equivalents, often achieve linear transfer performance
equal to that of end-to-end finetuning. We find relatively modest additional
downstream benefits from scaling model size, implying that scaling alone is not
sufficient to address our domain gap, and suggest that practitioners with
qualitatively different images might benefit more from in-domain adaption
followed by targeted downstream labelling.

我们首次系统研究了在星系图像上监督扩展定律的问题。使用了 840k 个星系图像和超过 1 亿个由 Galaxy Zoo 志愿者注释的图像，与 Imagenet-1K 的规模相当。我们发现添加注释的星系图像可以在所有架构和所有任务上提高性能，而添加可训练参数仅对某些任务有效。我们比较了仅在 ImageNet-12k 上进行预训练和在我们的星系图像上额外进行预训练的模型的下游性能。在 5 个科学相关的下游任务上，我们实现了平均相对误差率降低 31％的结果。我们的模型对标签的利用效率更高，而且通常实现了端到端微调的线性转移性能，与仅在 ImageNet-12k 上预训练的模型不同。我们发现仅通过扩展模型尺寸获得的额外下游效益相对较小，这意味着单纯的扩展不足以解决我们的领域差距问题，并建议具有定性不同的图像的从业者在领域内适应之后进行有针对性的下游分类。

星系图像的尺度律

Scaling Laws for Galaxy Images

The pretraining-finetuning paradigm has become the prevailing trend in modern
deep learning. In this work, we discover an intriguing linear phenomenon in
models that are initialized from a common pretrained checkpoint and finetuned
on different tasks, termed as Cross-Task Linearity (CTL). Specifically, if we
linearly interpolate the weights of two finetuned models, the features in the
weight-interpolated model are approximately equal to the linear interpolation
of features in two finetuned models at each layer. Such cross-task linearity
has not been noted in peer literature. We provide comprehensive empirical
evidence supporting that CTL consistently occurs for finetuned models that
start from the same pretrained checkpoint. We conjecture that in the
pretraining-finetuning paradigm, neural networks essentially function as linear
maps, mapping from the parameter space to the feature space. Based on this
viewpoint, our study unveils novel insights into explaining model
merging/editing, particularly by translating operations from the parameter
space to the feature space. Furthermore, we delve deeper into the underlying
factors for the emergence of CTL, emphasizing the impact of pretraining.

通过线性插值发现预训练 - 微调模型之间的交叉任务线性性，揭示神经网络在参数空间到特征空间的映射机制，提供了关于模型合并 / 编辑的新见解，并强调了预训练的影响因素。

预训练 - 微调模式中出现的任务交叉线性

Cross-Task Linearity Emerges in the Pretraining-Finetuning Paradigm

Previous studies observed that finetuned models may be better base models
than the vanilla pretrained model. Such a model, finetuned on some source
dataset, may provide a better starting point for a new finetuning process on a
desired target dataset. Here, we perform a systematic analysis of this
intertraining scheme, over a wide range of English classification tasks.
Surprisingly, our analysis suggests that the potential intertraining gain can
be analyzed independently for the target dataset under consideration, and for a
base model being considered as a starting point. This is in contrast to current
perception that the alignment between the target dataset and the source dataset
used to generate the base model is a major factor in determining intertraining
success. We analyze different aspects that contribute to each. Furthermore, we
leverage our analysis to propose a practical and efficient approach to
determine if and how to select a base model in real-world settings. Last, we
release an updating ranking of best models in the HuggingFace hub per
architecture this https URL.

通过对一系列英语分类任务的系统分析，本研究表明，可以独立分析考虑目标数据集和作为起点的基准模型的潜在相互训练收益，提出了一种实用且有效的方法来确定如何在实际情况下选择基准模型，同时在此给出了 HuggingFace Hub 每种架构最好的模型的最新排名列表。