Scaling up neural networks has been a key recipe to the success of large
language and vision models. However, in practice, up-scaled models can be
disproportionately costly in terms of computations, providing only marginal
improvements in performance; for example, EfficientViT-L3-384 achieves <2%
improvement on ImageNet-1K accuracy over the base L1-224 model, while requiring
$14\times$ more multiply-accumulate operations (MACs). In this paper, we
investigate scaling properties of popular families of neural networks for image
classification, and find that scaled-up models mostly help with "difficult"
samples. Decomposing the samples by difficulty, we develop a simple
model-agnostic two-pass Little-Big algorithm that first uses a light-weight
"little" model to make predictions of all samples, and only passes the
difficult ones for the "big" model to solve. Good little companion achieve
drastic MACs reduction for a wide variety of model families and scales. Without
loss of accuracy or modification of existing models, our Little-Big models
achieve MACs reductions of 76% for EfficientViT-L3-384, 81% for
EfficientNet-B7-600, 71% for DeiT3-L-384 on ImageNet-1K. Little-Big also speeds
up the InternImage-G-512 model by 62% while achieving 90% ImageNet-1K top-1
accuracy, serving both as a strong baseline and as a simple practical method
for large model compression.

使用轻量级的模型进行预测，然后仅通过困难样本对大模型进行求解的 Little-Big 算法能够显著减少计算量，实现对大型神经网络模型的压缩。