This paper discusses the feasibility of continuously training the CLIP model
through streaming data. Then, by tracking the directional changes of the
representation vectors in the continuously updated CLIP model, we explore and
summarize these spatial variations as Spatial Disorder (SD), which can be
divided into Intra-modal Rotation and Inter-modal Deviation. Moreover, we
demonstrate how intra-modal rotation and inter-modal deviation lead to a
performance decline for CLIP on cross-modal retrieval tasks in both empirically
and theoretically. To alleviate the spatial disorder, we propose a simple yet
effective continual learning framework Mod-X: Maintain off-diagonal
information-matriX. The experiments (in Section \ref{method}, \ref{experiments}
and Appendix \ref{Appendix_to_experiments}) on commonly used datasets with
different scales and scopes have illustrated the effectiveness of our method.

本文讨论了连续训练 CLIP 模型的可行性，并通过跟踪连续更新的 CLIP 模型中的表示向量的方向变化，将这些空间变化总结为空间扰动（SD），其可分为 Intra-modal Rotation 和 Inter-modal Deviation。此外，我们展示了 Intra-modal Rotation 和 Inter-modal Deviation 如何在经验上和理论上导致 CLIP 在跨模式检索任务中性能下降。为了缓解空间混乱，我们提出了一个简单而有效的连续学习框架 Mod-X：维护副对角线信息矩阵。在不同规模和范围的常用数据集（第ef {method} 节、ef {experiments} 节和附录ef {Appendix_to_experiments}）上的实验已经证明了我们方法的有效性。