Large-scale code generation models such as Codex and CodeT5 have achieved
impressive performance. However, libraries are upgraded or deprecated very
frequently and re-training large-scale language models is computationally
expensive. Therefore, Continual Learning (CL) is an important aspect that
remains underexplored in the code domain. In this paper, we introduce a
benchmark called CodeTask-CL that covers a wide range of tasks, including code
generation, translation, summarization, and refinement, with different input
and output programming languages. Next, on our CodeTask-CL benchmark, we
compare popular CL techniques from NLP and Vision domains. We find that
effective methods like Prompt Pooling (PP) suffer from catastrophic forgetting
due to the unstable training of the prompt selection mechanism caused by stark
distribution shifts in coding tasks. We address this issue with our proposed
method, Prompt Pooling with Teacher Forcing (PP-TF), that stabilizes training
by enforcing constraints on the prompt selection mechanism and leads to a
21.54% improvement over Prompt Pooling. Along with the benchmark, we establish
a training pipeline that can be used for CL on code models, which we believe
can motivate further development of CL methods for code models. Our code is
available at this https URL

本文介绍了一种名为 CodeTask-CL 的新颖基准测试，针对静态代码生成等多样的编码任务，比较了 NLP 和 Vision 领域中的几种不同的 Continual Learning 技术，提出了一种名为 Prompt Pooling with Teacher Forcing（PP-TF）的新的稳定训练的技术，取得了 21.54% 的改进效果，为大规模编码体系中的 Continual Learning 方法的进一步发展提供了支持。