Despite impressive advancements in multimodal compositional reasoning
approaches, they are still limited in their flexibility and efficiency by
processing fixed modality inputs while updating a lot of model parameters. This
paper tackles these critical challenges and proposes CREMA, an efficient and
modular modality-fusion framework for injecting any new modality into video
reasoning. We first augment multiple informative modalities (such as optical
flow, 3D point cloud, audio) from given videos without extra human annotation
by leveraging existing pre-trained models. Next, we introduce a query
transformer with multiple parameter-efficient modules associated with each
accessible modality. It projects diverse modality features to the LLM token
embedding space, allowing the model to integrate different data types for
response generation. Furthermore, we propose a fusion module designed to
compress multimodal queries, maintaining computational efficiency in the LLM
while combining additional modalities. We validate our method on video-3D,
video-audio, and video-language reasoning tasks and achieve better/equivalent
performance against strong multimodal LLMs, including BLIP-2, 3D-LLM, and
SeViLA while using 96% fewer trainable parameters. We provide extensive
analyses of CREMA, including the impact of each modality on reasoning domains,
the design of the fusion module, and example visualizations.

本文提出了一种高效的模态融合框架 CREMA，用于将任何新的模态注入视频推理，通过使用现有的预训练模型增强给定视频的多个信息模态，然后引入一个与每个可访问模态相关的多个参数高效模块的查询转换器，将不同的数据类型整合到响应产生的 LLM 令牌嵌入空间，同时提出了一个压缩多模态查询的融合模块，在维持 LLM 的计算效率的同时结合额外的模态，通过充分验证了在视频 - 3D、视频 - 音频和视频 - 语言推理任务上的性能，显示了优于其他强大的多模态 LLMs（包括 BLIP-2、3D-LLM 和 SeViLA）的表现，并使用了 96% 较少的可训练参数。

CREMA: 多模态组合视频推理的高效模块适应与融合

CREMA: Multimodal Compositional Video Reasoning via Efficient Modular  Adaptation and Fusion

Credal networks are a popular class of imprecise probabilistic graphical
models obtained as a Bayesian network generalization based on, so-called
credal, sets of probability mass functions. A Java library called CREMA has
been recently released to model, process and query credal networks. Despite the
NP-hardness of the (exact) task, a number of algorithms is available to
approximate credal network inferences. In this paper we present CREPO, an open
repository of synthetic credal networks, provided together with the exact
results of inference tasks on these models. A Python tool is also delivered to
load these data and interact with CREMA, thus making extremely easy to evaluate
and compare existing and novel inference algorithms. To demonstrate such
benchmarking scheme, we propose an approximate heuristic to be used inside
variable elimination schemes to keep a bound on the maximum number of vertices
generated during the combination step. A CREPO-based validation against
approximate procedures based on linearization and exact techniques performed in
CREMA is finally discussed.

本文提出 CREPO，一个开放的数据仓库，其中包含一组合成的 Credal networks 以及用于这些模型的精确推断任务的结果，还提供了一个 Python 工具用于加载这些数据和与 CREMA 交互，从而非常容易地评估和比较现有和新的推理算法。同时，本文提出了一种近似启发式算法，用于在变量消除方案中保持生成的顶点数量的最大边界，通过基于 CREPO 的验证，这种方法相对于基于线性化和精确技术的近似程序的效果更好。