In recent years, large-scale pre-trained multimodal models (LMM) generally
emerge to integrate the vision and language modalities, achieving considerable
success in various natural language processing and computer vision tasks. The
growing size of LMMs, however, results in a significant computational cost for
fine-tuning these models for downstream tasks. Hence, prompt-based interaction
strategy is studied to align modalities more efficiently. In this contex, we
propose a novel prompt-based multimodal interaction strategy inspired by human
memory strategy, namely Memory-Inspired Temporal Prompt Interaction (MITP). Our
proposed method involves in two stages as in human memory strategy: the
acquiring stage, and the consolidation and activation stage. We utilize
temporal prompts on intermediate layers to imitate the acquiring stage,
leverage similarity-based prompt interaction to imitate memory consolidation,
and employ prompt generation strategy to imitate memory activation. The main
strength of our paper is that we interact the prompt vectors on intermediate
layers to leverage sufficient information exchange between modalities, with
compressed trainable parameters and memory usage. We achieve competitive
results on several datasets with relatively small memory usage and 2.0M of
trainable parameters (about 1% of the pre-trained foundation model).

我们提出了一种受人类记忆策略启发的新型基于提示的多模态交互策略，通过在中间层上相互作用提示向量以实现足够的信息交换，同时具有压缩的可训练参数和内存使用，取得了具有竞争力的结果。