Much research in machine learning involves finding appropriate inductive
biases (e.g. convolutional neural networks, momentum-based optimizers,
transformers) to promote generalization on tasks. However, quantification of
the amount of inductive bias associated with these architectures and
hyperparameters has been limited. We propose a novel method for efficiently
computing the inductive bias required for generalization on a task with a fixed
training data budget; formally, this corresponds to the amount of information
required to specify well-generalizing models within a specific hypothesis space
of models. Our approach involves modeling the loss distribution of random
hypotheses drawn from a hypothesis space to estimate the required inductive
bias for a task relative to these hypotheses. Unlike prior work, our method
provides a direct estimate of inductive bias without using bounds and is
applicable to diverse hypothesis spaces. Moreover, we derive approximation
error bounds for our estimation approach in terms of the number of sampled
hypotheses. Consistent with prior results, our empirical results demonstrate
that higher dimensional tasks require greater inductive bias. We show that
relative to other expressive model classes, neural networks as a model class
encode large amounts of inductive bias. Furthermore, our measure quantifies the
relative difference in inductive bias between different neural network
architectures. Our proposed inductive bias metric provides an
information-theoretic interpretation of the benefits of specific model
architectures for certain tasks and provides a quantitative guide to developing
tasks requiring greater inductive bias, thereby encouraging the development of
more powerful inductive biases.

我们提出了一种计算归纳偏差的新方法，该方法涉及对来自假设空间的随机假设的损失分布进行建模，以估计相对于这些假设需要的归纳偏差，从而提供了一种信息理论解释特定模型架构对特定任务的好处的度量，并为开发需要更大归纳偏差的任务提供了定量指导，从而鼓励更强大的归纳偏差的发展。