In reinforcement learning the Q-values summarize the expected future rewards
that the agent will attain. However, they cannot capture the epistemic
uncertainty about those rewards. In this work we derive a new Bellman operator
with associated fixed point we call the `knowledge values'. These K-values
compress both the expected future rewards and the epistemic uncertainty into a
single value, so that high uncertainty, high reward, or both, can yield high
K-values. The key principle is to endow the agent with a risk-seeking utility
function that is carefully tuned to balance exploration and exploitation. When
the agent follows a Boltzmann policy over the K-values it yields a Bayes regret
bound of $\tilde O(L \sqrt{S A T})$, where $L$ is the time horizon, $S$ is the
total number of states, $A$ is the number of actions, and $T$ is the number of
elapsed timesteps. We show deep connections of this approach to the soft-max
and maximum-entropy strands of research in reinforcement learning.

本文中，我们利用一种新的 Bellman 算子和相应的固定点，称为‘知识价值’，将期望未来回报和认知不确定性压缩成一个单一的值，从而实现了一种风险追求效用函数，并通过 Boltzmann 策略生成贝叶斯风险边界。