The rise of large-scale multimodal models has paved the pathway for
groundbreaking advances in generative modeling and reasoning, unlocking
transformative applications in a variety of complex tasks. However, a pressing
question that remains is their genuine capability for stronger forms of
generalization, which has been largely underexplored in the multimodal setting.
Our study aims to address this by examining sequential compositional
generalization using \textsc{CompAct} (\underline{Comp}ositional
\underline{Act}ivities)\footnote{Project Page:
http://cyberiada.github.io/CompAct}, a carefully constructed,
perceptually grounded dataset set within a rich backdrop of egocentric kitchen
activity videos. Each instance in our dataset is represented with a combination
of raw video footage, naturally occurring sound, and crowd-sourced step-by-step
descriptions. More importantly, our setup ensures that the individual concepts
are consistently distributed across training and evaluation sets, while their
compositions are novel in the evaluation set. We conduct a comprehensive
assessment of several unimodal and multimodal models. Our findings reveal that
bi-modal and tri-modal models exhibit a clear edge over their text-only
counterparts. This highlights the importance of multimodality while charting a
trajectory for future research in this domain.

本研究通过研究多模态设置下的顺序组合泛化，检验了使用 CompAct（组合活动）数据集在丰富的自我中心厨房活动视频背景中。研究发现，双模态和三模态模型明显优于仅文本的对应模型，强调了多模态在未来研究中的重要性。