Large pre-trained models exhibit distinct and complementary capabilities
dependent on the data they are trained on. Language models such as GPT-3 are
capable of textual reasoning but cannot understand visual information, while
vision models such as DALL-E can generate photorealistic photos but fail to
understand complex language descriptions. In this work, we propose a unified
framework for composing ensembles of different pre-trained models -- combining
the strengths of each individual model to solve various multimodal problems in
a zero-shot manner. We use pre-trained models as "generators" or "scorers" and
compose them via closed-loop iterative consensus optimization. The generator
constructs proposals and the scorers iteratively provide feedback to refine the
generated result. Such closed-loop communication enables models to correct
errors caused by other models, significantly boosting performance on downstream
tasks, e.g. improving accuracy on grade school math problems by 7.5%, without
requiring any model finetuning. We demonstrate that consensus achieved by an
ensemble of scorers outperforms the feedback of a single scorer, by leveraging
the strengths of each expert model. Results show that the proposed method can
be used as a general purpose framework for a wide range of zero-shot multimodal
tasks, such as image generation, video question answering, mathematical
reasoning, and robotic manipulation. Project page:
this https URL.

本文提出了一种基于闭环迭代一致优化的框架，将不同的预训练模型组合在一起，以零样本方式解决各种多模态问题，该框架通过生成器和评分器的迭代反馈，使模型之间通过沟通逐渐纠正错误，形成一致性，从而显著提高下游任务的性能。