With recent advancements in large language models (LLMs), alignment has
emerged as an effective technique for keeping LLMs consensus with human intent.
Current methods primarily involve direct training through Supervised
Fine-tuning (SFT) or Reinforcement Learning from Human Feedback (RLHF), both of
which require substantial computational resources and extensive ground truth
data. This paper explores an efficient method for aligning black-box large
models using smaller models, introducing a model-agnostic and lightweight
Bayesian Persuasion Alignment framework. We formalize this problem as an
optimization of the signaling strategy from the small model's perspective. In
the persuasion process, the small model (Advisor) observes the information item
(i.e., state) and persuades large models (Receiver) to elicit improved
responses. The Receiver then generates a response based on the input, the
signal from the Advisor, and its updated belief about the information item.
Through training using our framework, we demonstrate that the Advisor can
significantly enhance the performance of various Receivers across a range of
tasks. We theoretically analyze our persuasion framework and provide an upper
bound on the Advisor's regret, confirming its effectiveness in learning the
optimal signaling strategy. Our Empirical results demonstrates that GPT-2 can
significantly improve the performance of various models, achieving an average
enhancement of 16.1% in mathematical reasoning ability and 13.7% in code
generation. We hope our work can provide an initial step toward rethinking the
alignment framework from the Bayesian Persuasion perspective.

通过使用较小的模型，引入一种模型不可知且轻量级的贝叶斯说服对齐框架，为对齐黑盒大模型提供高效方法。在使用该框架进行训练后，希望我们的工作能够提供从贝叶斯说服的视角重新审视对齐框架的初步步骤，并取得优于先前模型的性能提升。

基于贝叶斯说服的高效无模型对齐

Efficient Model-agnostic Alignment via Bayesian Persuasion

Strategic information disclosure, in its simplest form, considers a game
between an information provider (sender) who has access to some private
information that an information receiver is interested in. While the receiver
takes an action that affects the utilities of both players, the sender can
design information (or modify beliefs) of the receiver through signal
commitment, hence posing a Stackelberg game. However, obtaining a Stackelberg
equilibrium for this game traditionally requires the sender to have access to
the receiver's objective. In this work, we consider an online version of
information design where a sender interacts with a receiver of an unknown type
who is adversarially chosen at each round. Restricting attention to Gaussian
prior and quadratic costs for the sender and the receiver, we show that
$\mathcal{O}(\sqrt{T})$ regret is achievable with full information feedback,
where $T$ is the total number of interactions between the sender and the
receiver. Further, we propose a novel parametrization that allows the sender to
achieve $\mathcal{O}(\sqrt{T})$ regret for a general convex utility function.
We then consider the Bayesian Persuasion problem with an additional cost term
in the objective function, which penalizes signaling policies that are more
informative and obtain $\mathcal{O}(\log(T))$ regret. Finally, we establish a
sublinear regret bound for the partial information feedback setting and provide
simulations to support our theoretical results.

在本研究中，我们考虑了战略信息披露的不同方面，包括 Stackelberg 博弈、在线信息设计、贝叶斯说服和遗憾界限的分析。

学习如何策略性地披露信息

Learning How to Strategically Disclose Information

In Bayesian persuasion, an informed sender strategically discloses
information to a receiver so as to persuade them to undertake desirable
actions. Recently, a growing attention has been devoted to settings in which
sender and receivers interact sequentially. Recently, Markov persuasion
processes (MPPs) have been introduced to capture sequential scenarios where a
sender faces a stream of myopic receivers in a Markovian environment. The MPPs
studied so far in the literature suffer from issues that prevent them from
being fully operational in practice, e.g., they assume that the sender knows
receivers' rewards. We fix such issues by addressing MPPs where the sender has
no knowledge about the environment. We design a learning algorithm for the
sender, working with partial feedback. We prove that its regret with respect to
an optimal information-disclosure policy grows sublinearly in the number of
episodes, as it is the case for the loss in persuasiveness cumulated while
learning. Moreover, we provide a lower bound for our setting matching the
guarantees of our algorithm.

Bayesian persuasion and learning algorithms are used to address Markov persuasion processes where the sender has no prior knowledge, ensuring sublinear growth of regret in the number of episodes and matching the guarantees of the algorithm.

马尔可夫说服流程：从零开始学习说服

Markov Persuasion Processes: Learning to Persuade from Scratch

How can an informed sender persuade a receiver, having only limited
information about the receiver's beliefs? Motivated by research showing
generative AI can simulate economic agents, we initiate the study of
information design with an oracle. We assume the sender can learn more about
the receiver by querying this oracle, e.g., by simulating the receiver's
behavior. Aside from AI motivations such as general-purpose Large Language
Models (LLMs) and problem-specific machine learning models, alternate
motivations include customer surveys and querying a small pool of live users.
Specifically, we study Bayesian Persuasion where the sender has a
second-order prior over the receiver's beliefs. After a fixed number of queries
to an oracle to refine this prior, the sender commits to an information
structure. Upon receiving the message, the receiver takes a payoff-relevant
action maximizing her expected utility given her posterior beliefs. We design
polynomial-time querying algorithms that optimize the sender's expected utility
in this Bayesian Persuasion game. As a technical contribution, we show that
queries form partitions of the space of receiver beliefs that can be used to
quantify the sender's knowledge.

通过使用 oracle 查询来学习有关接收者的信仰，我们研究了信息设计与贝叶斯说服问题，设计了多项式时间查询算法来优化发送者的预期效用。

通过模拟进行算法说服：生成式人工智能时代的信息设计

Algorithmic Persuasion Through Simulation: Information Design in the Age  of Generative AI

We study a sequential decision making problem between a principal and an
agent with incomplete information on both sides. In this model, the principal
and the agent interact in a stochastic environment, and each is privy to
observations about the state not available to the other. The principal has the
power of commitment, both to elicit information from the agent and to provide
signals about her own information. The principal and the agent communicate
their signals to each other, and select their actions independently based on
this communication. Each player receives a payoff based on the state and their
joint actions, and the environment moves to a new state. The interaction
continues over a finite time horizon, and both players act to optimize their
own total payoffs over the horizon. Our model encompasses as special cases
stochastic games of incomplete information and POMDPs, as well as sequential
Bayesian persuasion and mechanism design problems. We study both computation of
optimal policies and learning in our setting. While the general problems are
computationally intractable, we study algorithmic solutions under a conditional
independence assumption on the underlying state-observation distributions. We
present an polynomial-time algorithm to compute the principal's optimal policy
up to an additive approximation. Additionally, we show an efficient learning
algorithm in the case where the transition probabilities are not known
beforehand. The algorithm guarantees sublinear regret for both players.

本文研究了一个在信息不完备的情况下，委托人和代理人之间的顺序决策问题。

具有通信的顺序主 - 代理问题：高效计算与学习

Sequential Principal-Agent Problems with Communication: Efficient  Computation and Learning

We consider a novel dynamic pricing and learning setting where in addition to
setting prices of products in sequential rounds, the seller also ex-ante
commits to 'advertising schemes'. That is, in the beginning of each round the
seller can decide what kind of signal they will provide to the buyer about the
product's quality upon realization. Using the popular Bayesian persuasion
framework to model the effect of these signals on the buyers' valuation and
purchase responses, we formulate the problem of finding an optimal design of
the advertising scheme along with a pricing scheme that maximizes the seller's
expected revenue. Without any apriori knowledge of the buyers' demand function,
our goal is to design an online algorithm that can use past purchase responses
to adaptively learn the optimal pricing and advertising strategy. We study the
regret of the algorithm when compared to the optimal clairvoyant price and
advertising scheme.
Our main result is a computationally efficient online algorithm that achieves
an $O(T^{2/3}(m\log T)^{1/3})$ regret bound when the valuation function is
linear in the product quality. Here $m$ is the cardinality of the discrete
product quality domain and $T$ is the time horizon. This result requires some
natural monotonicity and Lipschitz assumptions on the valuation function, but
no Lipschitz or smoothness assumption on the buyers' demand function. For
constant $m$, our result matches the regret lower bound for dynamic pricing
within logarithmic factors, which is a special case of our problem. We also
obtain several improved results for the widely considered special case of
additive valuations, including an $\tilde{O}(T^{2/3})$ regret bound independent
of $m$ when $m\le T^{1/3}$.

该研究探讨在线广告和动态定价设计的问题，使用贝叶斯说服模型来研究信号对买家定价和购买决策的影响，提出了一种具有低后悔率的在线算法。