We study off-policy learning (OPL) of contextual bandit policies in large
discrete action spaces where existing methods -- most of which rely crucially
on reward-regression models or importance-weighted policy gradients -- fail due
to excessive bias or variance. To overcome these issues in OPL, we propose a
novel two-stage algorithm, called Policy Optimization via Two-Stage Policy
Decomposition (POTEC). It leverages clustering in the action space and learns
two different policies via policy- and regression-based approaches,
respectively. In particular, we derive a novel low-variance gradient estimator
that enables to learn a first-stage policy for cluster selection efficiently
via a policy-based approach. To select a specific action within the cluster
sampled by the first-stage policy, POTEC uses a second-stage policy derived
from a regression-based approach within each cluster. We show that a local
correctness condition, which only requires that the regression model preserves
the relative expected reward differences of the actions within each cluster,
ensures that our policy-gradient estimator is unbiased and the second-stage
policy is optimal. We also show that POTEC provides a strict generalization of
policy- and regression-based approaches and their associated assumptions.
Comprehensive experiments demonstrate that POTEC provides substantial
improvements in OPL effectiveness particularly in large and structured action
spaces.

通过提出一种新的两阶段算法 POTEC，利用动作空间的聚类和基于策略以及回归的方法，研究了大规模离散动作空间中上下文赌博机政策的离线学习问题，结果显示 POTEC 在特别是大且结构化的动作空间中显著提高了离线学习的效果。

POTEC：基于两阶段策略分解的大动作空间离线学习

POTEC: Off-Policy Learning for Large Action Spaces via Two-Stage Policy  Decomposition

Algorithmic decision making process now affects many aspects of our lives.
Standard tools for machine learning, such as classification and regression, are
subject to the bias in data, and thus direct application of such off-the-shelf
tools could lead to a specific group being unfairly discriminated. Removing
sensitive attributes of data does not solve this problem because a
\textit{disparate impact} can arise when non-sensitive attributes and sensitive
attributes are correlated. Here, we study a fair machine learning algorithm
that avoids such a disparate impact when making a decision. Inspired by the
two-stage least squares method that is widely used in the field of economics,
we propose a two-stage algorithm that removes bias in the training data. The
proposed algorithm is conceptually simple. Unlike most of existing fair
algorithms that are designed for classification tasks, the proposed method is
able to (i) deal with regression tasks, (ii) combine explanatory attributes to
remove reverse discrimination, and (iii) deal with numerical sensitive
attributes. The performance and fairness of the proposed algorithm are
evaluated in simulations with synthetic and real-world datasets.

论文提出了一种基于两阶段最小二乘法的公平机器学习算法，能够有效处理数据中敏感属性与非敏感属性相关的不公平问题， 其独特之处在于该算法针对回归模型的不公平问题也能做出有效处理，并且能够解决数字敏感属性的问题，经实验验证表现卓越。