The Artificial Intelligence field is flooded with optimisation methods. In
this paper, we change the focus to developing modelling methods with the aim of
getting us closer to Artificial General Intelligence. To do so, we propose a
novel way to interpret reality as an information source, that is later
translated into a computational framework able to capture and represent such
information. This framework is able to build elements of classical cognitive
architectures, like Long Term Memory and Working Memory, starting from a simple
primitive that only processes Spatial Distributed Representations. Moreover, it
achieves such level of verticality in a seamless scalable hierarchical way.

通过将现实解读为信息源，并将其转化为计算框架，我们提出了一种新颖的建模方法，从而接近于人工通用智能。该框架可以以流动可扩展的分层方式构建经典的认知架构元素，如长期记忆和工作记忆。

从表现到认知体系结构：一个可扩展的框架

From Manifestations to Cognitive Architectures: a Scalable Framework

Training neural networks with first order optimisation methods is at the core
of the empirical success of deep learning. The scale of initialisation is a
crucial factor, as small initialisations are generally associated to a feature
learning regime, for which gradient descent is implicitly biased towards simple
solutions. This work provides a general and quantitative description of the
early alignment phase, originally introduced by Maennel et al. (2018) . For
small initialisation and one hidden ReLU layer networks, the early stage of the
training dynamics leads to an alignment of the neurons towards key directions.
This alignment induces a sparse representation of the network, which is
directly related to the implicit bias of gradient flow at convergence. This
sparsity inducing alignment however comes at the expense of difficulties in
minimising the training objective: we also provide a simple data example for
which overparameterised networks fail to converge towards global minima and
only converge to a spurious stationary point instead.

使用一阶优化方法训练神经网络是深度学习实验成功的核心。在小的初始化下，训练动力学的早期阶段会导致神经元朝关键方向对齐，从而引发网络的稀疏表示，这与渐近梯度流的隐式偏差相关。然而，这种稀疏诱导的对齐会导致在最小化训练目标上面临困难，我们还提供了一个简单的数据示例，说明过度参数化的网络无法收敛到全局极小值点，而只能收敛到一个虚假的稳定点。

两层网络训练中的早期对齐是双刃剑

Early alignment in two-layer networks training is a two-edged sword

Hamiltonian Monte Carlo (HMC) is a popular Markov chain Monte Carlo (MCMC)
algorithm that generates proposals for a Metropolis-Hastings algorithm by
simulating the dynamics of a Hamiltonian system. However, HMC is sensitive to
large time discretizations and performs poorly if there is a mismatch between
the spatial geometry of the target distribution and the scales of the momentum
distribution. In particular the mass matrix of HMC is hard to tune well. In
order to alleviate these problems we propose relativistic Hamiltonian Monte
Carlo, a version of HMC based on relativistic dynamics that introduce a maximum
velocity on particles. We also derive stochastic gradient versions of the
algorithm and show that the resulting algorithms bear interesting relationships
to gradient clipping, RMSprop, Adagrad and Adam, popular optimisation methods
in deep learning. Based on this, we develop relativistic stochastic gradient
descent by taking the zero-temperature limit of relativistic stochastic
gradient Hamiltonian Monte Carlo. In experiments we show that the relativistic
algorithms perform better than classical Newtonian variants and Adam.

提出了基于相对论动力学的哈密顿蒙特卡罗方法，通过引入粒子的最大速度解决哈密顿蒙特卡罗在大时间离散化和空间几何不匹配时的性能问题，并开发了基于此的相对论随机梯度下降算法，与深度学习中的优化方法如梯度截断、RMSprop、Adagrad 和 Adam 有趣的关系，实验表明这种算法比经典牛顿变体和 Adam 表现更好。