Large Multimodal Models (LMMs) exhibit remarkable multi-tasking ability by learning mixed datasets jointly. However, novel tasks would be encountered sequentially in dynamic world, and continually fine-tuning LMMs often leads to performance degrades. To handle the challenges of catastrophic forgetting, existing methods leverage data replay or model expansion, both of which are not specially developed for LMMs and have their inherent limitations. In this paper, we propose a novel dual-modality guided prompt learning framework (ModalPrompt) tailored for multimodal continual learning to effectively learn new tasks while alleviating forgetting of previous knowledge. Concretely, we learn prototype prompts for each task and exploit efficient prompt selection for task identifiers and prompt fusion for knowledge transfer based on image-text supervision. Extensive experiments demonstrate the superiority of our approach, e.g., ModalPrompt achieves +20% performance gain on LMMs continual learning benchmarks with $\times$ 1.42 inference speed refraining from growing training cost in proportion to the number of tasks. The code will be made publically available.

本研究解决了大型多模态模型在动态环境中持续学习新任务时的灾难性遗忘问题。提出了一种专为多模态持续学习设计的双模态引导提示学习框架（ModalPrompt），通过任务原型提示学习和高效提示选择，显著提高了新任务的学习效果，同时有效减轻了对先前知识的遗忘。实验结果表明，该方法在持续学习基准上实现了+20%的性能提升，并保持了较高的推理速度。

ModalPrompt：用于大型多模态模型持续学习的双模态引导提示