In continual learning (CL), a learner is faced with a sequence of tasks,
arriving one after the other, and the goal is to remember all the tasks once
the continual learning experience is finished. The prior art in CL uses
episodic memory, parameter regularization or extensible network structures to
reduce interference among tasks, but in the end, all the approaches learn
different tasks in a joint vector space. We believe this invariably leads to
interference among different tasks. We propose to learn tasks in different
(low-rank) vector subspaces that are kept orthogonal to each other in order to
minimize interference. Further, to keep the gradients of different tasks coming
from these subspaces orthogonal to each other, we learn isometric mappings by
posing network training as an optimization problem over the Stiefel manifold.
To the best of our understanding, we report, for the first time, strong results
over experience-replay baseline with and without memory on standard
classification benchmarks in continual learning. The code is made publicly
available.

通过在不同的向量子空间中学习任务并在其中保持正交性来最小化干扰，提出了一种新的连续学习的方法，并证明了其对标准分类基准测试的优越性能。

低秩正交子空间下的持续学习

Continual Learning in Low-rank Orthogonal Subspaces

We examine a class of embeddings based on structured random matrices with
orthogonal rows which can be applied in many machine learning applications
including dimensionality reduction and kernel approximation. For both the
Johnson-Lindenstrauss transform and the angular kernel, we show that we can
select matrices yielding guaranteed improved performance in accuracy and/or
speed compared to earlier methods. We introduce matrices with complex entries
which give significant further accuracy improvement. We provide geometric and
Markov chain-based perspectives to help understand the benefits, and empirical
results which suggest that the approach is helpful in a wider range of
applications.

本文使用基于结构化随机矩阵和正交行的嵌入，分析了其在机器学习中的应用，包括降维和核逼近，同时通过实证结果证明了该方法在各种机器学习应用中具有明显提高精度和 / 或速度的效果。

结构化随机正交嵌入的不合理有效性

The Unreasonable Effectiveness of Structured Random Orthogonal  Embeddings

Gaussian processes models are widely adopted for
nonparameteric/semi-parametric modeling. Identifiability issues occur when the
mean model contains polynomials with unknown coefficients. Though resulting
prediction is unaffected, this leads to poor estimation of the coefficients in
the mean model, and thus the estimated mean model loses interpretability. This
paper introduces a new Gaussian process model whose stochastic part is
orthogonal to the mean part to address this issue. This paper also discusses
applications to multi-fidelity simulations using data examples.

本文提出一种新的高斯过程模型，其随机部分与均值部分正交，以解决均值模型包含具有未知系数的多项式时出现的可识别性问题，并探讨其在多保真度模拟中的应用。