The chain-of-thought technique has been received well in multi-modal tasks.
It is a step-by-step linear reasoning process that adjusts the length of the
chain to improve the performance of generated prompts. However, human thought
processes are predominantly non-linear, as they encompass multiple aspects
simultaneously and employ dynamic adjustment and updating mechanisms.
Therefore, we propose a novel Aggregation-Graph-of-Thought (AGoT) mechanism for
soft-prompt tuning in multi-modal representation learning. The proposed AGoT
models the human thought process not only as a chain but also models each step
as a reasoning aggregation graph to cope with the overlooked multiple aspects
of thinking in single-step reasoning. This turns the entire reasoning process
into prompt aggregation and prompt flow operations. Experiments show that our
multi-modal model enhanced with AGoT soft-prompting achieves good results in
several tasks such as text-image retrieval, visual question answering, and
image recognition. In addition, we demonstrate that it has good domain
generalization performance due to better reasoning.

我们提出了一种新颖的 Aggregation-Graph-of-Thought (AGoT) 机制，用于在多模态表示学习中进行软提示调整，该机制将人类思维过程建模为链式加以思维图，并且通过聚合和流动操作的提示将整个思考过程转化为优化问题，实验证明，我们的多模态模型采用了 AGoT 软提示的方法在文本 - 图像检索、视觉问答和图像识别等任务上获得了良好的结果，同时也因为更好的推理能力在领域泛化方面表现出色。

多模态表示学习的思维图软提示

Soft-Prompting with Graph-of-Thought for Multi-modal Representation  Learning

The large language models have achieved superior performance on various
natural language tasks. One major drawback of such approaches is they are
resource-intensive in fine-tuning new datasets. Soft-prompt tuning presents a
resource-efficient solution to fine-tune the pre-trained language models (PLMs)
while keeping their weight frozen. Existing soft prompt methods mainly focus on
designing the input-independent prompts that steer the model to fit the domain
of the new dataset. Those methods often ignore the fine-grained information
about the task and context of the text. In this paper, we propose a multi-level
prompt tuning (MPrompt) method for machine reading comprehension. It utilizes
prompts at task-specific, domain-specific, and context-specific levels to
enhance the comprehension of input semantics at different granularities. We
also propose an independence constraint to steer each domain-specific prompt to
focus on information within its domain to avoid redundancy. Moreover, we
present a prompt generator that incorporates context-related knowledge in the
prompt generation to enhance contextual relevancy. We conducted extensive
experiments on 12 benchmarks of various QA formats and achieved an average
improvement of 1.94\% over the state-of-the-art methods.

该研究提出了一种多级提示调优方法来增强机器阅读理解，通过利用任务特定、领域特定和上下文相关的提示，提高了输入语义在不同层次上的理解能力。同时，引入了独立性约束来避免冗余，并利用上下文相关知识生成合适的提示。在各种问答格式的 12 个基准测试中进行了广泛实验，相比最先进的方法，平均改进率达到了 1.94%。

MPrompt: 多级提示调优在机器阅读理解中的探索

MPrompt: Exploring Multi-level Prompt Tuning for Machine Reading  Comprehension

Prompting large language models has gained immense popularity in recent years
due to the advantage of producing good results even without the need for
labelled data. However, this requires prompt tuning to get optimal prompts that
lead to better model performances. In this paper, we explore the use of
soft-prompt tuning on sentiment classification task to quantify the biases of
large language models (LLMs) such as Open Pre-trained Transformers (OPT) and
Galactica language model. Since these models are trained on real-world data
that could be prone to bias toward certain groups of populations, it is
important to identify these underlying issues. Using soft-prompts to evaluate
bias gives us the extra advantage of avoiding the human-bias injection that can
be caused by manually designed prompts. We check the model biases on different
sensitive attributes using the group fairness (bias) and find interesting bias
patterns. Since LLMs have been used in the industry in various applications, it
is crucial to identify the biases before deploying these models in practice. We
open-source our pipeline and encourage industry researchers to adapt our work
to their use cases.

本文探讨在情感分类任务上使用软提示调整识别大型语言模型（LLMs）如 OPT 和 Galactica 语言模型中的偏见，使用群体公平性（偏见）来检查模型偏见，并发现有趣的偏见模式，以便在实践中部署这些模型之前，识别这些偏见。