We consider estimation under scenarios where the signals of interest exhibit change of characteristics over time. In particular, we consider the continual learning problem where different tasks, e.g., data with different distributions, arrive sequentially and the aim is to perform well on the newly arrived task without performance degradation on the previously seen tasks. In contrast to the continual learning literature focusing on the centralized setting, we investigate the problem from a distributed estimation perspective. We consider the well-established distributed learning algorithm COCOA, which distributes the model parameters and the corresponding features over the network. We provide exact analytical characterization for the generalization error of COCOA under continual learning for linear regression in a range of scenarios, where overparameterization is of particular interest. These analytical results characterize how the generalization error depends on the network structure, the task similarity and the number of tasks, and show how these dependencies are intertwined. In particular, our results show that the generalization error can be significantly reduced by adjusting the network size, where the most favorable network size depends on task similarity and the number of tasks. We present numerical results verifying the theoretical analysis and illustrate the continual learning performance of COCOA with a digit classification task.

我们从分布式估计的角度研究在信号特征随时间变化的情况下的估计问题，具体而言，我们考虑在不同任务（例如具有不同分布的数据）连续到达的情况下，如何在新到达的任务上表现良好同时不降低先前任务的性能，我们研究了分布式学习算法COCOA在线性回归下的通用化错误，并在一系列情景中提供了确切的分析特征，其中特别关注了过度拟合问题。这些分析结果表明通用化错误如何取决于网络结构、任务相似性和任务数，并展示了这些依赖关系如何交织在一起。特别是，我们的结果表明通过调整网络规模可以显著降低通用化错误，而最有利的网络规模取决于任务相似性和任务数。我们还提供了数值结果来验证理论分析，并通过一个数字分类任务展示了COCOA在连续学习中的性能。

高维线性回归中的分布式连续学习与 CoCoA