BriefGPT.xyz
Nov, 2023
大型多模态模型的持续指令调优
Continual Instruction Tuning for Large Multimodal Models
HTML
PDF
Jinghan He, Haiyun Guo, Ming Tang, Jinqiao Wang
TL;DR
本文研究了指令调整在持续学习大型多模型 (LMMs) 中的应用,并发现持续调整过程中仍然存在灾难性遗忘现象,但多任务联合指令调整可以改善遗忘问题,此外,基于任务相似性的正则化和模型扩展方法在持续学习中表现良好。
Abstract
instruction tuning
is now a widely adopted approach to aligning large multimodal models (LMMs) to follow human intent. It unifies the data format of
vision-language tasks
, enabling
→