We introduce Venn Diagram (VD) Prompting, an innovative prompting technique
which allows Large Language Models (LLMs) to combine and synthesize information
across complex, diverse and long-context documents in knowledge-intensive
question-answering tasks. Generating answers from multiple documents involves
numerous steps to extract relevant and unique information and amalgamate it
into a cohesive response. To improve the quality of the final answer, multiple
LLM calls or pretrained models are used to perform different tasks such as
summarization, reorganization and customization. The approach covered in the
paper focuses on replacing the multi-step strategy via a single LLM call using
VD prompting. Our proposed technique also aims to eliminate the inherent
position bias in the LLMs, enhancing consistency in answers by removing
sensitivity to the sequence of input information. It overcomes the challenge of
inconsistency traditionally associated with varying input sequences. We also
explore the practical applications of the VD prompt based on our examination of
the prompt's outcomes. In the experiments performed on four public benchmark
question-answering datasets, VD prompting continually matches or surpasses the
performance of a meticulously crafted instruction prompt which adheres to
optimal guidelines and practices.

介绍了一种名为 Venn Diagram Prompting 的创新提示技术，允许大型语言模型在知识密集型问答任务中将复杂、多样和长文档的信息结合和综合起来，从而提高最终答案的质量和一致性。

促进文本理解的 Venn 图提示：扩展支持效应

Venn Diagram Prompting : Accelerating Comprehension with Scaffolding  Effect

Large language models (LLMs) such as GPT-4 have exhibited remarkable
performance in a variety of tasks, but this strong performance often comes with
the high expense of using paid API services. In this paper, we are motivated to
study building an LLM cascade to save the cost of using LLMs, particularly for
performing reasoning (e.g., mathematical, causal) tasks. Our cascade pipeline
follows the intuition that simpler questions can be addressed by a weaker but
more affordable LLM, whereas only the challenging questions necessitate the
stronger and more expensive LLM. To realize this decision-making, we consider
the "answer consistency" of the weaker LLM as a signal of the question
difficulty and propose several methods for the answer sampling and consistency
checking, including one leveraging a mixture of two thought representations
(i.e., Chain-of-Thought and Program-of-Thought). Through experiments on six
reasoning benchmark datasets, with GPT-3.5-turbo and GPT-4 being the weaker and
stronger LLMs, respectively, we demonstrate that our proposed LLM cascades can
achieve performance comparable to using solely the stronger LLM but require
only 40% of its cost.

通过构建 LLM 级联模型来实现节约成本，特别是在推理任务中的使用，通过应用弱一些但更便宜的 LLM 来解决简单问题，只有复杂问题才需要更强大且更昂贵的 LLM，实现这种决策的关键是通过检查较弱 LLM 的 “答案一致性” 来判断问题的难度，并提出了几种答案采样和一致性检查的方法，其中有一个利用了两种思考表示（Chain-of-Thought 和 Program-of-Thought）的混合方法，在六个推理基准数据集上的实验证明，我们的 LLM 级联模型可以达到与单独使用更强大的 LLM 相当的性能，但仅需 40% 的成本。