Despite impressive advancements in multimodal compositional reasoning
approaches, they are still limited in their flexibility and efficiency by
processing fixed modality inputs while updating a lot of model parameters. This
paper tackles these critical challenges and proposes CREMA, an efficient and
modular modality-fusion framework for injecting any new modality into video
reasoning. We first augment multiple informative modalities (such as optical
flow, 3D point cloud, audio) from given videos without extra human annotation
by leveraging existing pre-trained models. Next, we introduce a query
transformer with multiple parameter-efficient modules associated with each
accessible modality. It projects diverse modality features to the LLM token
embedding space, allowing the model to integrate different data types for
response generation. Furthermore, we propose a fusion module designed to
compress multimodal queries, maintaining computational efficiency in the LLM
while combining additional modalities. We validate our method on video-3D,
video-audio, and video-language reasoning tasks and achieve better/equivalent
performance against strong multimodal LLMs, including BLIP-2, 3D-LLM, and
SeViLA while using 96% fewer trainable parameters. We provide extensive
analyses of CREMA, including the impact of each modality on reasoning domains,
the design of the fusion module, and example visualizations.

本文提出了一种高效的模态融合框架 CREMA，用于将任何新的模态注入视频推理，通过使用现有的预训练模型增强给定视频的多个信息模态，然后引入一个与每个可访问模态相关的多个参数高效模块的查询转换器，将不同的数据类型整合到响应产生的 LLM 令牌嵌入空间，同时提出了一个压缩多模态查询的融合模块，在维持 LLM 的计算效率的同时结合额外的模态，通过充分验证了在视频 - 3D、视频 - 音频和视频 - 语言推理任务上的性能，显示了优于其他强大的多模态 LLMs（包括 BLIP-2、3D-LLM 和 SeViLA）的表现，并使用了 96% 较少的可训练参数。

CREMA: 多模态组合视频推理的高效模块适应与融合

CREMA: Multimodal Compositional Video Reasoning via Efficient Modular  Adaptation and Fusion

Large language models (LLMs) have been widely used in various applications
but are known to suffer from issues related to untruthfulness and toxicity.
While parameter-efficient modules (PEMs) have demonstrated their effectiveness
in equipping models with new skills, leveraging PEMs for deficiency unlearning
remains underexplored. In this work, we propose a PEMs operation approach,
namely Extraction-before-Subtraction (Ext-Sub), to enhance the truthfulness and
detoxification of LLMs through the integration of ``expert'' PEM and
``anti-expert'' PEM. Remarkably, even anti-expert PEM possess valuable
capabilities due to their proficiency in generating fabricated content, which
necessitates language modeling and logical narrative competence. Rather than
merely negating the parameters, our approach involves extracting and
eliminating solely the deficiency capability within anti-expert PEM while
preserving the general capabilities. To evaluate the effectiveness of our
approach in terms of truthfulness and detoxification, we conduct extensive
experiments on LLMs, encompassing additional abilities such as language
modeling and mathematical reasoning. Our empirical results demonstrate that our
approach effectively improves truthfulness and detoxification, while largely
preserving the fundamental abilities of LLMs.

通过整合 “专家” 和 “反专家” 参数，我们提出了一种称为 “Ext-Sub” 的参数有效模块操作方法，以提高大型语言模型的真实性和去毒性，并在保留通用能力的同时提取和消除 “反专家” 参数内的缺陷能力。通过对语言模型和数学推理等额外能力进行广泛实验，我们的实证结果表明我们的方法有效地改善了大型语言模型的真实性和去毒性。