Although statistical learning theory provides a robust framework to
understand supervised learning, many theoretical aspects of deep learning
remain unclear, in particular how different architectures may lead to inductive
bias when trained using gradient based methods. The goal of these lectures is
to provide an overview of some of the main questions that arise when attempting
to understand deep learning from a learning theory perspective. After a brief
reminder on statistical learning theory and stochastic optimization, we discuss
implicit bias in the context of benign overfitting. We then move to a general
description of the mirror descent algorithm, showing how we may go back and
forth between a parameter space and the corresponding function space for a
given learning problem, as well as how the geometry of the learning problem may
be represented by a metric tensor. Building on this framework, we provide a
detailed study of the implicit bias of gradient descent on linear diagonal
networks for various regression tasks, showing how the loss function, scale of
parameters at initialization and depth of the network may lead to various forms
of implicit bias, in particular transitioning between kernel or feature
learning.

从学习理论的角度来理解深度学习时，我们讨论了一些主要问题，包括统计学习理论、随机优化、梯度下降对线性对角网络的隐式偏差。

将统计学习理论应用于深度学习

Applying statistical learning theory to deep learning

Many approaches for optimizing decision making systems rely on gradient based
methods requiring informative feedback from the environment. However, in the
case where such feedback is sparse or uninformative, such approaches may result
in poor performance. Derivative-free approaches such as Bayesian Optimization
mitigate the dependency on the quality of gradient feedback, but are known to
scale poorly in the high-dimension setting of complex decision making systems.
This problem is exacerbated if the system requires interactions between several
actors cooperating to accomplish a shared goal. To address the dimensionality
challenge, we propose a compact multi-layered architecture modeling the
dynamics of actor interactions through the concept of role. Additionally, we
introduce Hessian-aware Bayesian Optimization to efficiently optimize the
multi-layered architecture parameterized by a large number of parameters.
Experimental results demonstrate that our method (HA-GP-UCB) works effectively
on several benchmarks under resource constraints and malformed feedback
settings.

优化决策系统的许多方法依赖于基于梯度的方法，然而在缺乏信息或反馈不明确的情况下，这些方法可能导致性能不佳。为了解决高维复杂决策系统中的挑战，我们提出了一个紧凑的多层架构模型来建模角色之间的互动，同时引入了 Hessian-aware Bayesian Optimization 以高效优化参数化了大量参数的多层架构。实验证明，我们的方法（HA-GP-UCB）在资源限制和反馈错误设置下能够有效工作。

基于海森矩阵感知的贝叶斯优化在决策系统中的应用

Hessian-Aware Bayesian Optimization for Decision Making Systems

While gradient based methods are ubiquitous in machine learning, selecting
the right step size often requires "hyperparameter tuning". This is because
backtracking procedures like Armijo's rule depend on quality evaluations in
every step, which are not available in a stochastic context. Since optimization
schemes can be motivated using Taylor approximations, we replace the Taylor
approximation with the conditional expectation (the best $L^2$ estimator) and
propose "Random Function Descent" (RFD). Under light assumptions common in
Bayesian optimization, we prove that RFD is identical to gradient descent, but
with calculable step sizes, even in a stochastic context. We beat untuned Adam
in synthetic benchmarks. To close the performance gap to tuned Adam, we propose
a heuristic extension competitive with tuned Adam.

在机器学习中普及的基于梯度的方法需要 “超参数调整”，因为 Armijo 规则等回溯过程需要每一步的质量评估并且不适用于随机优化。本文提出了一种名为 “随机函数下降” 的优化方法，证明在贝叶斯优化场景下，RFD 与梯度下降是相同的，但具有可计算的步长，在合成基准测试中超越了未调整的 Adam 方法，并提出一种启发式扩展，使其与调整后的 Adam 相当。