Computer vision models suffer from a phenomenon known as catastrophic
forgetting when learning novel concepts from continuously shifting training
data. Typical solutions for this continual learning problem require extensive
rehearsal of previously seen data, which increases memory costs and may violate
data privacy. Recently, the emergence of large-scale pre-trained vision
transformer models has enabled prompting approaches as an alternative to
data-rehearsal. These approaches rely on a key-query mechanism to generate
prompts and have been found to be highly resistant to catastrophic forgetting
in the well-established rehearsal-free continual learning setting. However, the
key mechanism of these methods is not trained end-to-end with the task
sequence. Our experiments show that this leads to a reduction in their
plasticity, hence sacrificing new task accuracy, and inability to benefit from
expanded parameter capacity. We instead propose to learn a set of prompt
components which are assembled with input-conditioned weights to produce
input-conditioned prompts, resulting in a novel attention-based end-to-end
key-query scheme. Our experiments show that we outperform the current SOTA
method DualPrompt on established benchmarks by as much as 5.4% in average
accuracy. We also outperform the state of art by as much as 6.6% accuracy on a
continual learning benchmark which contains both class-incremental and
domain-incremental task shifts, corresponding to many practical settings.

提出了一种注意力机制的端到端关键查询方案，该方案使用输入为条件的权重来组装预置组件以生成输入为条件的提示，以解决在不断变化的训练数据中进行学习时出现的过度遗忘问题，并在多个基准测试中大幅提高了准确度。