The theory of reinforcement learning has focused on two fundamental problems:
achieving low regret, and identifying $\epsilon$-optimal policies. While a
simple reduction allows one to apply a low-regret algorithm to obtain an
$\epsilon$-optimal policy and achieve the worst-case optimal rate, it is
unknown whether low-regret algorithms can obtain the instance-optimal rate for
policy identification. We show this is not possible -- there exists a
fundamental tradeoff between achieving low regret and identifying an
$\epsilon$-optimal policy at the instance-optimal rate.
Motivated by our negative finding, we propose a new measure of
instance-dependent sample complexity for PAC tabular reinforcement learning
which explicitly accounts for the attainable state visitation distributions in
the underlying MDP. We then propose and analyze a novel, planning-based
algorithm which attains this sample complexity -- yielding a complexity which
scales with the suboptimality gaps and the "reachability" of a state. We show
our algorithm is nearly minimax optimal, and on several examples that our
instance-dependent sample complexity offers significant improvements over
worst-case bounds.

研究提出了一种新的针对 PAC 表格强化学习的实例相关样本复杂度的计算方法，并设计了一种能够达到该样本复杂度的规划算法，该算法几乎是极小值最优的，且在多个实例上展现出比最坏情况界限更显著的改进。