Foundational vision-language models have shown impressive performance on
various downstream tasks. Yet, there is still a pressing need to update these
models later as new tasks or domains become available. Ongoing Continual
Learning (CL) research provides techniques to overcome catastrophic forgetting
of previous information when new knowledge is acquired. To date, CL techniques
focus only on the supervised training sessions. This results in significant
forgetting yielding inferior performance to even the prior model zero shot
performance. In this work, we argue that test-time data hold great information
that can be leveraged in a self supervised manner to refresh the model's memory
of previous learned tasks and hence greatly reduce forgetting at no extra
labelling cost. We study how unsupervised data can be employed online to
improve models' performance on prior tasks upon encountering representative
samples. We propose a simple yet effective student-teacher model with gradient
based sparse parameters updates and show significant performance improvements
and reduction in forgetting, which could alleviate the role of an offline
episodic memory/experience replay buffer.

通过利用测试数据进行自我监督，以减少遗忘，并提出了基于梯度稀疏参数更新的简单但有效的学生 - 教师模型，进而显著改善了模型的性能和减少了遗忘。

在连续学习中利用测试时间数据控制遗忘

Controlling Forgetting with Test-Time Data in Continual Learning

Deep neural networks often have a huge number of parameters, which posts
challenges in deployment in application scenarios with limited memory and
computation capacity. Knowledge distillation is one approach to derive compact
models from bigger ones. However, it has been observed that a converged heavy
teacher model is strongly constrained for learning a compact student network
and could make the optimization subject to poor local optima. In this paper, we
propose ProKT, a new model-agnostic method by projecting the supervision
signals of a teacher model into the student's parameter space. Such projection
is implemented by decomposing the training objective into local intermediate
targets with an approximate mirror descent technique. The proposed method could
be less sensitive with the quirks during optimization which could result in a
better local optimum. Experiments on both image and text datasets show that our
proposed ProKT consistently achieves superior performance compared to other
existing knowledge distillation methods.

本文提出了一种名为 ProKT 的知识蒸馏方法，通过将教师模型的监督信号投影到学生参数空间中，从而在优化过程中实现了更好的局部最优解，实验结果表明，与其他现有的知识蒸馏方法相比，ProKT 在图像和文本数据集上都展现出了卓越的性能。

跟随自己的道路：一种渐进式的知识蒸馏方法

Follow Your Path: a Progressive Method for Knowledge Distillation

Knowledge distillation is typically conducted by training a small model (the
student) to mimic a large and cumbersome model (the teacher). The idea is to
compress the knowledge from the teacher by using its output probabilities as
soft-labels to optimize the student. However, when the teacher is considerably
large, there is no guarantee that the internal knowledge of the teacher will be
transferred into the student; even if the student closely matches the
soft-labels, its internal representations may be considerably different. This
internal mismatch can undermine the generalization capabilities originally
intended to be transferred from the teacher to the student. In this paper, we
propose to distill the internal representations of a large model such as BERT
into a simplified version of it. We formulate two ways to distill such
representations and various algorithms to conduct the distillation. We
experiment with datasets from the GLUE benchmark and consistently show that
adding knowledge distillation from internal representations is a more powerful
method than only using soft-label distillation.

本文提出了通过知识蒸馏从内部表示来压缩 BERT 这样的大型模型，并阐述了两种从内部表示中提取知识的方法和不同算法的实验。结论是，与仅使用软标签蒸馏相比，从内部表示来蒸馏是更强大的方法。