The Stackelberg game model, where a leader commits to a strategy and the
follower best responds, has found widespread application, particularly to
security problems. In the security setting, the goal is for the leader to
compute an optimal strategy to commit to, in order to protect some asset. In
many of these applications, the parameters of the follower utility model are
not known with certainty. Distributionally-robust optimization addresses this
issue by allowing a distribution over possible model parameters, where this
distribution comes from a set of possible distributions. The goal is to
maximize the expected utility with respect to the worst-case distribution. We
initiate the study of distributionally-robust models for computing the optimal
strategy to commit to. We consider the case of normal-form games with
uncertainty about the follower utility model. Our main theoretical result is to
show that a distributionally-robust Stackelberg equilibrium always exists
across a wide array of uncertainty models. For the case of a finite set of
possible follower utility functions we present two algorithms to compute a
distributionally-robust strong Stackelberg equilibrium (DRSSE) using
mathematical programs. Next, in the general case where there is an infinite
number of possible follower utility functions and the uncertainty is
represented by a Wasserstein ball around a finitely-supported nominal
distribution, we give an incremental mixed-integer-programming-based algorithm
for computing the optimal distributionally-robust strategy. Experiments
substantiate the tractability of our algorithm on a classical Stackelberg game,
showing that our approach scales to medium-sized games.

本文研究了基于分布鲁棒优化的 Stackelberg 游戏模型，特别是在不确定下属效用模型的情况下，寻找最优策略以保护某些资产。我们的研究表明，在广泛的不确定模型下，分布鲁棒 Stackelberg 均衡始终存在。当存在无限个下属效用函数且不确定性由支持有限名义分布周围的 Wasserstein 球表示时，我们提出了一种基于混合整数规划的增量算法来计算最优的分布鲁棒策略。实验结果显示，我们的方法可扩展到中等规模的 Stackelberg 游戏，从而验证了算法的可行性。

计算最优分布式鲁棒策略

Computing the optimal distributionally-robust strategy to commit to

Robust Reinforcement Learning (RL) focuses on improving performances under
model errors or adversarial attacks, which facilitates the real-life deployment
of RL agents. Robust Adversarial Reinforcement Learning (RARL) is one of the
most popular frameworks for robust RL. However, most of the existing literature
models RARL as a zero-sum simultaneous game with Nash equilibrium as the
solution concept, which could overlook the sequential nature of RL deployments,
produce overly conservative agents, and induce training instability. In this
paper, we introduce a novel hierarchical formulation of robust RL - a
general-sum Stackelberg game model called RRL-Stack - to formalize the
sequential nature and provide extra flexibility for robust training. We develop
the Stackelberg Policy Gradient algorithm to solve RRL-Stack, leveraging the
Stackelberg learning dynamics by considering the adversary's response. Our
method generates challenging yet solvable adversarial environments which
benefit RL agents' robust learning. Our algorithm demonstrates better training
stability and robustness against different testing conditions in the
single-agent robotics control and multi-agent highway merging tasks.

本文介绍了一种用于强化学习的 Stackelberg 游戏模型 ——RRL-Stack，旨在提供额外的鲁棒性训练和解决目前 RL 训练中存在的过度保守智能及训练不稳定等问题，并提出了一种基于 Stackelberg Policy Gradient 算法的解决方案，在单一和多智能体任务中展现更好的训练稳定性和鲁棒性。

通过适应性规则对抗训练实现史塔克伯格博弈的坚韧强化学习

Robust Reinforcement Learning as a Stackelberg Game via Adaptively-Regularized Adversarial Training

For the sake of protecting data privacy and due to the rapid development of
mobile devices, e.g., powerful central processing unit (CPU) and nascent neural
processing unit (NPU), collaborative machine learning on mobile devices, e.g.,
federated learning, has been envisioned as a new AI approach with broad
application prospects. However, the learning process of the existing federated
learning platforms rely on the direct communication between the model owner,
e.g., central cloud or edge server, and the mobile devices for transferring the
model update. Such a direct communication may be energy inefficient or even
unavailable in mobile environments. In this paper, we consider adopting the
relay network to construct a cooperative communication platform for supporting
model update transfer and trading. In the system, the mobile devices generate
model updates based on their training data. The model updates are then
forwarded to the model owner through the cooperative relay network. The model
owner enjoys the learning service provided by the mobile devices. In return,
the mobile devices charge the model owner certain prices. Due to the coupled
interference of wireless transmission among the mobile devices that use the
same relay node, the rational mobile devices have to choose their relay nodes
as well as deciding on their transmission powers. Thus, we formulate a
Stackelberg game model to investigate the interaction among the mobile devices
and that between the mobile devices and the model owner. The Stackelberg
equilibrium is investigated by capitalizing on the exterior point method.
Moreover, we provide a series of insightful analytical and numerical results on
the equilibrium of the Stackelberg game.

采用中继网络构建合作通信平台支持模型更新传输和交易的博弈模型研究。

联合定价与合作中继通信用于联邦学习

Joint Service Pricing and Cooperative Relay Communication for Federated  Learning

There has been significant recent interest in game-theoretic approaches to
security, with much of the recent research focused on utilizing the
leader-follower Stackelberg game model. Among the major applications are the
ARMOR program deployed at LAX Airport and the IRIS program in use by the US
Federal Air Marshals (FAMS). The foundational assumption for using Stackelberg
games is that security forces (leaders), acting first, commit to a randomized
strategy; while their adversaries (followers) choose their best response after
surveillance of this randomized strategy. Yet, in many situations, a leader may
face uncertainty about the follower's surveillance capability. Previous work
fails to address how a leader should compute her strategy given such
uncertainty. We provide five contributions in the context of a general class of
security games. First, we show that the Nash equilibria in security games are
interchangeable, thus alleviating the equilibrium selection problem. Second,
under a natural restriction on security games, any Stackelberg strategy is also
a Nash equilibrium strategy; and furthermore, the solution is unique in a class
of security games of which ARMOR is a key exemplar. Third, when faced with a
follower that can attack multiple targets, many of these properties no longer
hold. Fourth, we show experimentally that in most (but not all) games where the
restriction does not hold, the Stackelberg strategy is still a Nash equilibrium
strategy, but this is no longer true when the attacker can attack multiple
targets. Finally, as a possible direction for future research, we propose an
extensive-form game model that makes the defender's uncertainty about the
attacker's ability to observe explicit.

本文研究了使用 Stackelberg 博弈模型在安全领域的应用。在 leader-follower 模型中，一个 leader 可能面临着关于 follower 的监视能力的不确定性问题。本研究提出了五个方案和扩展形式博弈模型，以解决这个问题并阐明 Nash 均衡的特点。