The online advertising market, with its thousands of auctions run per second,
presents a daunting challenge for advertisers who wish to optimize their spend
under a budget constraint. Thus, advertising platforms typically provide
automated agents to their customers, which act on their behalf to bid for
impression opportunities in real time at scale. Because these proxy agents are
owned by the platform but use advertiser funds to operate, there is a strong
practical need to balance reliability and explainability of the agent with
optimizing power. We propose a generalizable approach to optimizing bidding
policies in production environments by learning from real data using offline
reinforcement learning. This approach can be used to optimize any
differentiable base policy (practically, a heuristic policy based on principles
which the advertiser can easily understand), and only requires data generated
by the base policy itself. We use a hybrid agent architecture that combines
arbitrary base policies with deep neural networks, where only the optimized
base policy parameters are eventually deployed, and the neural network part is
discarded after training. We demonstrate that such an architecture achieves
statistically significant performance gains in both simulated and at-scale
production bidding environments. Our approach does not incur additional
infrastructure, safety, or explainability costs, as it directly optimizes
parameters of existing production routines without replacing them with black
box-style models like neural networks.

通过离线强化学习从真实数据中学习，我们提出了一种通用方法来优化生产环境中的竞价策略，这种方法可以用于优化任何可微分的基础策略，并且只需要由基础策略自身生成的数据。我们使用混合代理架构将任意基础策略与深度神经网络结合起来，在训练后仅部署优化的基础策略参数，而神经网络部分被丢弃。我们证明这样的架构在模拟和生产竞价环境中都能取得统计上显著的性能提升。我们的方法不会产生额外的基础设施、安全或可解释性成本，因为它直接优化现有生产流程的参数，而不是用类似神经网络的黑箱模型替换它们。