The chain-of-thought technique has been received well in multi-modal tasks.
It is a step-by-step linear reasoning process that adjusts the length of the
chain to improve the performance of generated prompts. However, human thought
processes are predominantly non-linear, as they encompass multiple aspects
simultaneously and employ dynamic adjustment and updating mechanisms.
Therefore, we propose a novel Aggregation-Graph-of-Thought (AGoT) mechanism for
soft-prompt tuning in multi-modal representation learning. The proposed AGoT
models the human thought process not only as a chain but also models each step
as a reasoning aggregation graph to cope with the overlooked multiple aspects
of thinking in single-step reasoning. This turns the entire reasoning process
into prompt aggregation and prompt flow operations. Experiments show that our
multi-modal model enhanced with AGoT soft-prompting achieves good results in
several tasks such as text-image retrieval, visual question answering, and
image recognition. In addition, we demonstrate that it has good domain
generalization performance due to better reasoning.

我们提出了一种新颖的 Aggregation-Graph-of-Thought (AGoT) 机制，用于在多模态表示学习中进行软提示调整，该机制将人类思维过程建模为链式加以思维图，并且通过聚合和流动操作的提示将整个思考过程转化为优化问题，实验证明，我们的多模态模型采用了 AGoT 软提示的方法在文本 - 图像检索、视觉问答和图像识别等任务上获得了良好的结果，同时也因为更好的推理能力在领域泛化方面表现出色。