Recent advances in self-supervised learning and neural network scaling have
enabled the creation of large models -- known as foundation models -- which can
be easily adapted to a wide range of downstream tasks. The current paradigm for
comparing foundation models involves benchmarking them with aggregate metrics
on various curated datasets. Unfortunately, this method of model comparison is
heavily dependent on the choice of metric, which makes it unsuitable for
situations where the ideal metric is either not obvious or unavailable. In this
work, we present a metric-free methodology for comparing foundation models via
their embedding space geometry. Our methodology is grounded in random graph
theory, and facilitates both pointwise and multi-model comparison. Further, we
demonstrate how our framework can be used to induce a manifold of models
equipped with a distance function that correlates strongly with several
downstream metrics.

本文提出了一种不需要度量的基于嵌入空间几何的方法来比较基础模型，并展示了该框架如何用于诱导一组带有距离函数的模型流型，其与多个下游度量强相关。

使用数据核心比较基础模型

Comparing Foundation Models using Data Kernels

Neural network scaling has been critical for improving the model quality in
many real-world machine learning applications with vast amounts of training
data and compute. Although this trend of scaling is affirmed to be a sure-fire
approach for better model quality, there are challenges on the path such as the
computation cost, ease of programming, and efficient implementation on parallel
devices. GShard is a module composed of a set of lightweight annotation APIs
and an extension to the XLA compiler. It provides an elegant way to express a
wide range of parallel computation patterns with minimal changes to the
existing model code. GShard enabled us to scale up multilingual neural machine
translation Transformer model with Sparsely-Gated Mixture-of-Experts beyond 600
billion parameters using automatic sharding. We demonstrate that such a giant
model can efficiently be trained on 2048 TPU v3 accelerators in 4 days to
achieve far superior quality for translation from 100 languages to English
compared to the prior art.

利用 GShard 能够高效地扩展神经网络规模，支持多语言机器翻译，使得能够在 4 天内用 TPU v3 加速器训练 600 亿参数的模型，并且在从 100 种语言翻译成英语方面的性能要明显优于之前的工作。