Incremental gradient methods and incremental proximal methods are a
fundamental class of optimization algorithms used for solving finite sum
problems, broadly studied in the literature. Yet, when it comes to their
convergence guarantees, nonasymptotic (first-order or proximal) oracle
complexity bounds have been obtained fairly recently, almost exclusively
applying to the average iterate. Motivated by applications in continual
learning, we obtain the first convergence guarantees for the last iterate of
both incremental gradient and incremental proximal methods, in general convex
smooth (for both) and convex Lipschitz (for the proximal variants) settings.
Our oracle complexity bounds for the last iterate nearly match (i.e., match up
to a square-root-log or a log factor) the best known oracle complexity bounds
for the average iterate, for both classes of methods. We further obtain
generalizations of our results to weighted averaging of the iterates with
increasing weights, which can be seen as interpolating between the last iterate
and the average iterate guarantees. Additionally, we discuss how our results
can be generalized to variants of studied incremental methods with permuted
ordering of updates. Our results generalize last iterate guarantees for
incremental methods compared to state of the art, as such results were
previously known only for overparameterized linear models, which correspond to
convex quadratic problems with infinitely many solutions.

基于应用于连续学习中的动机，我们首次获得了增量梯度和增量近端方法的最后迭代的收敛性保证，适用于一般凸平滑以及凸 Lipschitz 设置，同时几乎与平均迭代的最佳收敛性保证相匹配。

增量方法的最后迭代收敛及其在持续学习中的应用

Last Iterate Convergence of Incremental Methods and Applications in  Continual Learning

We study a generalization of boosting to the multiclass setting. We introduce
a weak learning condition for multiclass classification that captures the
original notion of weak learnability as being "slightly better than random
guessing". We give a simple and efficient boosting algorithm, that does not
require realizability assumptions and its sample and oracle complexity bounds
are independent of the number of classes.
In addition, we utilize our new boosting technique in several theoretical
applications within the context of List PAC Learning. First, we establish an
equivalence to weak PAC learning. Furthermore, we present a new result on
boosting for list learners, as well as provide a novel proof for the
characterization of multiclass PAC learning and List PAC learning. Notably, our
technique gives rise to a simplified analysis, and also implies an improved
error bound for large list sizes, compared to previous results.

将增强学习推广到多类别情景中，引入了适用于多类别分类的弱学习条件，提出了一种简单高效的增强学习算法，并在列表式 PAC 学习的背景下应用其新的增强技术，证明了其可行性和可靠性。