Continual learning (CL) with Vision-Language Models (VLMs) has overcome the
constraints of traditional CL, which only focuses on previously encountered
classes. During the CL of VLMs, we need not only to prevent the catastrophic
forgetting on incrementally learned knowledge but also to preserve the
zero-shot ability of VLMs. However, existing methods require additional
reference datasets to maintain such zero-shot ability and rely on
domain-identity hints to classify images across different domains. In this
study, we propose Regression-based Analytic Incremental Learning (RAIL), which
utilizes a recursive ridge regression-based adapter to learn from a sequence of
domains in a non-forgetting manner and decouple the cross-domain correlations
by projecting features to a higher-dimensional space. Cooperating with a
training-free fusion module, RAIL absolutely preserves the VLM's zero-shot
ability on unseen domains without any reference data. Additionally, we
introduce Cross-domain Task-Agnostic Incremental Learning (X-TAIL) setting. In
this setting, a CL learner is required to incrementally learn from multiple
domains and classify test images from both seen and unseen domains without any
domain-identity hint. We theoretically prove RAIL's absolute memorization on
incrementally learned domains. Experiment results affirm RAIL's
state-of-the-art performance in both X-TAIL and existing Multi-domain
Task-Incremental Learning settings. The code will be released upon acceptance.

利用回归分析的增量学习和跨领域任务无关增量学习方法，保持视觉 - 语言模型的零 - shot 能力和在多领域下的性能。