Bayesian reinforcement learning (RL) offers a principled and elegant approach
for sequential decision making under uncertainty. Most notably, Bayesian agents
do not face an exploration/exploitation dilemma, a major pathology of
frequentist methods. A key challenge for Bayesian RL is the computational
complexity of learning Bayes-optimal policies, which is only tractable in toy
domains. In this paper we propose a novel model-free approach to address this
challenge. Rather than modelling uncertainty in high-dimensional state
transition distributions as model-based approaches do, we model uncertainty in
a one-dimensional Bellman operator. Our theoretical analysis reveals that
existing model-free approaches either do not propagate epistemic uncertainty
through the MDP or optimise over a set of contextual policies instead of all
history-conditioned policies. Both approximations yield policies that can be
arbitrarily Bayes-suboptimal. To overcome these issues, we introduce the
Bayesian exploration network (BEN) which uses normalising flows to model both
the aleatoric uncertainty (via density estimation) and epistemic uncertainty
(via variational inference) in the Bellman operator. In the limit of complete
optimisation, BEN learns true Bayes-optimal policies, but like in variational
expectation-maximisation, partial optimisation renders our approach tractable.
Empirical results demonstrate that BEN can learn true Bayes-optimal policies in
tasks where existing model-free approaches fail.

贝叶斯强化学习在面对不确定性的顺序决策问题中提供了一种原则性和优雅的方法，但其主要挑战是在高维状态转移分布中建模不确定性的计算复杂性。本文提出了一种新颖的无模型方法来解决这个挑战，通过在一维贝尔曼算子中建模不确定性，引入贝叶斯探索网络 (BEN)，通过正态化流来建模贝尔曼算子中的不确定性，并通过变分推断来建模知识性不确定性，实验结果表明，BEN 可以在现有的无模型方法失败的任务中学习到真正的贝叶斯最优策略。