We study the relationship between catastrophic forgetting and properties of
task sequences. In particular, given a sequence of tasks, we would like to
understand which properties of this sequence influence the error rates of
continual learning algorithms trained on the sequence. To this end, we propose
a new procedure that makes use of recent developments in task space modeling as
well as correlation analysis to specify and analyze the properties we are
interested in. As an application, we apply our procedure to study two
properties of a task sequence: (1) total complexity and (2) sequential
heterogeneity. We show that error rates are strongly and positively correlated
to a task sequence's total complexity for some state-of-the-art algorithms. We
also show that, surprisingly, the error rates have no or even negative
correlations in some cases to sequential heterogeneity. Our findings suggest
directions for improving continual learning benchmarks and methods.

本文研究了灾难性遗忘和任务序列属性之间的关系，尤其是给定任务序列，我们希望了解这个序列的哪些属性会影响在该序列上训练的永续学习算法的错误率。我们提出了一种新的程序，利用最近在任务空间建模和相关性分析方面的发展来指定和分析我们感兴趣的属性。作为一个应用，我们将我们的程序应用于研究任务序列的两个属性：“总复杂度” 和 “顺序异质性”。我们发现，对于一些最先进的算法，错误率与任务序列的总复杂度强烈且正相关；令人惊讶的是，在某些情况下，错误率与顺序异质性没有或甚至具有负相关性。我们的发现为改进永续学习基准和方法提供了方向。