We introduce a novel capacity measure 2sED for statistical models based on
the effective dimension. The new quantity provably bounds the generalization
error under mild assumptions on the model. Furthermore, simulations on standard
data sets and popular model architectures show that 2sED correlates well with
the training error. For Markovian models, we show how to efficiently
approximate 2sED from below through a layerwise iterative approach, which
allows us to tackle deep learning models with a large number of parameters.
Simulation results suggest that the approximation is good for different
prominent models and data sets.

我们引入了一种基于有效维度的统计模型的新容量度量 2sED。这个新数量在模型的温和假设下能证明地限制了泛化误差。此外，对标准数据集和流行的模型架构的模拟结果表明，2sED 与训练误差具有良好的相关性。对于马尔可夫模型，我们展示了如何通过分层迭代方法从下方高效地逼近 2sED，这使我们能够处理具有大量参数的深度学习模型。模拟结果表明，这种逼近适用于不同的突出模型和数据集。

深度学习模型的双尺度复杂度度量

A Two-Scale Complexity Measure for Deep Learning Models

In reinforcement learning, state representations are used to tractably deal
with large problem spaces. State representations serve both to approximate the
value function with few parameters, but also to generalize to newly encountered
states. Their features may be learned implicitly (as part of a neural network)
or explicitly (for example, the successor representation of
\citet{dayan1993improving}). While the approximation properties of
representations are reasonably well-understood, a precise characterization of
how and when these representations generalize is lacking. In this work, we
address this gap and provide an informative bound on the generalization error
arising from a specific state representation. This bound is based on the notion
of effective dimension which measures the degree to which knowing the value at
one state informs the value at other states. Our bound applies to any state
representation and quantifies the natural tension between representations that
generalize well and those that approximate well. We complement our theoretical
results with an empirical survey of classic representation learning methods
from the literature and results on the Arcade Learning Environment, and find
that the generalization behaviour of learned representations is well-explained
by their effective dimension.

本研究探讨了强化学习中状态表示的泛化行为及具体的有效维度边界，并通过文献调研及案例分析验证了所提出的有效维度理论。

强化学习中表示的泛化

On the Generalization of Representations in Reinforcement Learning

Fault-tolerant quantum computers offer the promise of dramatically improving
machine learning through speed-ups in computation or improved model
scalability. In the near-term, however, the benefits of quantum machine
learning are not so clear. Understanding expressibility and trainability of
quantum models-and quantum neural networks in particular-requires further
investigation. In this work, we use tools from information geometry to define a
notion of expressibility for quantum and classical models. The effective
dimension, which depends on the Fisher information, is used to prove a novel
generalisation bound and establish a robust measure of expressibility. We show
that quantum neural networks are able to achieve a significantly better
effective dimension than comparable classical neural networks. To then assess
the trainability of quantum models, we connect the Fisher information spectrum
to barren plateaus, the problem of vanishing gradients. Importantly, certain
quantum neural networks can show resilience to this phenomenon and train faster
than classical models due to their favourable optimisation landscapes, captured
by a more evenly spread Fisher information spectrum. Our work is the first to
demonstrate that well-designed quantum neural networks offer an advantage over
classical neural networks through a higher effective dimension and faster
training ability, which we verify on real quantum hardware.

本文使用信息几何工具，定义了量子和经典神经网络的表达能力，将有效维度作为证明表达能力的新广义边界，并建立了一个强大的表达能力度量。我们展示了量子神经网络能够比相似的经典神经网络实现更好的有效维度，并因其更均匀分布的 Fisher 信息谱而显示了对枯燥的高原问题具有适应性并具有快速的训练能力。我们的工作是第一个通过更高的有效维度和更快的训练能力证明精心设计的量子神经网络优于经典神经网络的，并在实际量子硬件上进行了验证。

量子神经网络的威力

The power of quantum neural networks

Gibbs-ERM learning is a natural idealized model of learning with stochastic
optimization algorithms (such as Stochastic Gradient Langevin Dynamics and
---to some extent--- Stochastic Gradient Descent), while it also arises in
other contexts, including PAC-Bayesian theory, and sampling mechanisms. In this
work we study the excess risk suffered by a Gibbs-ERM learner that uses
non-convex, regularized empirical risk with the goal to understand the
interplay between the data-generating distribution and learning in large
hypothesis spaces. Our main results are distribution-dependent upper bounds on
several notions of excess risk. We show that, in all cases, the
distribution-dependent excess risk is essentially controlled by the effective
dimension $\mathrm{tr}\left(\boldsymbol{H}^{\star} (\boldsymbol{H}^{\star} +
\lambda \boldsymbol{I})^{-1}\right)$ of the problem, where
$\boldsymbol{H}^{\star}$ is the Hessian matrix of the risk at a local minimum.
This is a well-established notion of effective dimension appearing in several
previous works, including the analyses of SGD and ridge regression, but ours is
the first work that brings this dimension to the analysis of learning using
Gibbs densities. The distribution-dependent view we advocate here improves upon
earlier results of Raginsky et al. (2017), and can yield much tighter bounds
depending on the interplay between the data-generating distribution and the
loss function. The first part of our analysis focuses on the localized excess
risk in the vicinity of a fixed local minimizer. This result is then extended
to bounds on the global excess risk, by characterizing probabilities of local
minima (and their complement) under Gibbs densities, a results which might be
of independent interest.

本研究通过对 Gibbs-ERM 学习的实验，得出使用正则化经验风险的 Gibbs-ERM 学习器所遭受的过度风险并受数据生成分布和大假设空间影响的分布依赖上界受有效维度控制，这对于研究技术至关重要。