Despite the general capabilities of pre-trained large language models (LLMs),
they still need further adaptation to better serve practical applications. In
this paper, we demonstrate the interchangeability of three popular and distinct
adaptation tools: parameter updating, reward modeling, and in-context
prompting. This interchangeability establishes a triangular framework with six
transformation directions, each of which facilitates a variety of applications.
Our work offers a holistic view that unifies numerous existing studies and
suggests potential research directions. We envision our work as a useful
roadmap for future research on LLMs.

我们展示了三种常用的适应工具的互换性，分别为参数更新、奖励建模和上下文提示，形成了一个三角形框架，建立了六个转换方向，每个转换方向都有助于实现各种应用。我们的工作为现有研究提供了一个整体观点，并提出了潜在的研究方向。我们希望我们的工作成为未来 LLM 研究的有用路线图。

关于奖励模型、参数更新和上下文提示的变换

On the Transformations across Reward Model, Parameter Update, and  In-Context Prompt

In-context prompting in large language models (LLMs) has become a prevalent
approach to improve zero-shot capabilities, but this idea is less explored in
the vision domain. Existing visual prompting methods focus on referring
segmentation to segment the most relevant object, falling short of addressing
many generic vision tasks like open-set segmentation and detection. In this
paper, we introduce a universal visual in-context prompting framework for both
tasks. In particular, we build on top of an encoder-decoder architecture, and
develop a versatile prompt encoder to support a variety of prompts like
strokes, boxes, and points. We further enhance it to take an arbitrary number
of reference image segments as the context. Our extensive explorations show
that the proposed visual in-context prompting elicits extraordinary referring
and generic segmentation capabilities to refer and detect, yielding competitive
performance to close-set in-domain datasets and showing promising results on
many open-set segmentation datasets. By joint training on COCO and SA-1B, our
model achieves $57.7$ PQ on COCO and $23.2$ PQ on ADE20K. Code will be
available at this https URL

本文介绍了一种通用的视觉上下文提示框架，以支持涂鸦、方框和点等各种提示类型，并进一步改进以支持任意数量的上下文。通过在 COCO 和 SA-1B 上进行联合训练，我们的模型在 COCO 上达到 57.7 PQ，在 ADE20K 上达到 23.2 PQ。

视觉背景提示

Visual In-Context Prompting

This study investigates the application of Large Language Models (LLMs),
specifically GPT-4, within Astronomy. We employ in-context prompting, supplying
the model with up to 1000 papers from the NASA Astrophysics Data System, to
explore the extent to which performance can be improved by immersing the model
in domain-specific literature. Our findings point towards a substantial boost
in hypothesis generation when using in-context prompting, a benefit that is
further accentuated by adversarial prompting. We illustrate how adversarial
prompting empowers GPT-4 to extract essential details from a vast knowledge
base to produce meaningful hypotheses, signaling an innovative step towards
employing LLMs for scientific research in Astronomy.

该研究探讨了在天文学中应用大型语言模型（LLMs），尤其是 GPT-4，通过上下文提示，将模型提供给多达 1000 篇来自 NASA 天体物理学数据系统的论文，探索通过将模型浸入特定域的文献中可能提高性能的程度。我们的发现表明，在使用上下文提示时，假设生成可以得到实质性的提升，并且通过对抗性提示进一步强调了这一优势。我们展示了对抗性提示如何使 GPT-4 从庞大的知识库中提取关键细节以产生有意义的假设，这标志着将 LLMs 用于天文学科学研究的创新步骤。