The cost of hyperparameter tuning in deep learning has been rising with model
sizes, prompting practitioners to find new tuning methods using a proxy of
smaller networks. One such proposal uses $\mu$P parameterized networks, where
the optimal hyperparameters for small width networks transfer to networks with
arbitrarily large width. However, in this scheme, hyperparameters do not
transfer across depths. As a remedy, we study residual networks with a residual
branch scale of $1/\sqrt{\text{depth}}$ in combination with the $\mu$P
parameterization. We provide experiments demonstrating that residual
architectures including convolutional ResNets and Vision Transformers trained
with this parameterization exhibit transfer of optimal hyperparameters across
width and depth on CIFAR-10 and ImageNet. Furthermore, our empirical findings
are supported and motivated by theory. Using recent developments in the
dynamical mean field theory (DMFT) description of neural network learning
dynamics, we show that this parameterization of ResNets admits a well-defined
feature learning joint infinite-width and infinite-depth limit and show
convergence of finite-size network dynamics towards this limit.

使用深度学习调参中的参数化方法，研究残差网络的超参数如何在不同宽度和深度的网络中传递，并证明实验和理论结果的一致性。

残差网络中的深度超参数转移：动态和尺度限制

Depthwise Hyperparameter Transfer in Residual Networks: Dynamics and  Scaling Limit

Though deep reinforcement learning agents have achieved unprecedented success
in recent years, their learned policies can be brittle, failing to generalize
to even slight modifications of their environments or unfamiliar situations.
The black-box nature of the neural network learning dynamics makes it
impossible to audit trained deep agents and recover from such failures. In this
paper, we propose a novel representation and learning approach to capture
environment dynamics without using neural networks. It originates from the
observation that, in games designed for people, the effect of an action can
often be perceived in the form of local changes in consecutive visual
observations. Our algorithm is designed to extract such vision-based changes
and condense them into a set of action-dependent descriptive rules, which we
call ''visual rewrite rules'' (VRRs). We also present preliminary results from
a VRR agent that can explore, expand its rule set, and solve a game via
planning with its learned VRR world model. In several classical games, our
non-deep agent demonstrates superior performance, extreme sample efficiency,
and robust generalization ability compared with several mainstream deep agents.

本文提出了一种使用规则来捕获游戏动态的深度强化学习代理方法，该方法不使用神经网络，且在多个传统游戏中表现出优异的性能，极高的样本效率和强大的泛化能力。