Deep neural networks achieve human-like performance on a variety of
perceptual and decision-making tasks. However, networks perform poorly when
confronted with changing tasks or goals, and broadly fail to match the
flexibility and robustness of human intelligence. Here, we develop a
mathematical and algorithmic framework that enables flexible and continuous
training of neural networks on a range of objectives by constructing path
connected sets of networks that achieve equivalent functional performance on a
given machine learning task. We view the weight space of a neural network as a
curved Riemannian manifold and move a network along a functionally invariant
path in weight space while searching for networks that satisfy secondary
objectives. A path-sampling algorithm trains computer vision and natural
language processing networks with millions of weight parameters to learn a
series of classification tasks without performance loss while accommodating
secondary objectives including network sparsification, incremental task
learning, and increased adversarial robustness. Broadly, we conceptualize a
neural network as a mathematical object that can be iteratively transformed
into distinct configurations by the path-sampling algorithm to define a
sub-manifold of networks that can be harnessed to achieve user goals.

通过构建具有功能等效性的网络的路径连接集合，将神经网络的权重空间视为弯曲的黎曼流形并沿着功能不变路径将网络移动，同时寻找满足次要目标的网络，通过路径抽样算法，能够训练具有数百万权重参数的神经网络来学习一系列分类任务而不会损失性能，同时适应包括网络稀疏化，增量任务学习和增加对抗强度在内的次要目标。