Two central paradigms have emerged in the reinforcement learning (RL)
community: online RL and offline RL. In the online RL setting, the agent has no
prior knowledge of the environment, and must interact with it in order to find
an $\epsilon$-optimal policy. In the offline RL setting, the learner instead
has access to a fixed dataset to learn from, but is unable to otherwise
interact with the environment, and must obtain the best policy it can from this
offline data. Practical scenarios often motivate an intermediate setting: if we
have some set of offline data and, in addition, may also interact with the
environment, how can we best use the offline data to minimize the number of
online interactions necessary to learn an $\epsilon$-optimal policy?
In this work, we consider this setting, which we call the \textsf{FineTuneRL}
setting, for MDPs with linear structure. We characterize the necessary number
of online samples needed in this setting given access to some offline dataset,
and develop an algorithm, \textsc{FTPedel}, which is provably optimal. We show
through an explicit example that combining offline data with online
interactions can lead to a provable improvement over either purely offline or
purely online RL. Finally, our results illustrate the distinction between
\emph{verifiable} learning, the typical setting considered in online RL, and
\emph{unverifiable} learning, the setting often considered in offline RL, and
show that there is a formal separation between these regimes.

本文考虑了具有线性结构的 MDPs 的 FineTuneRL 设置，并开发了一种称为 FTPedel 的算法，用于结合脱机数据和在线 RL 以改进学习表现，结果证明了在线样本数的必要性以及在线 RL 和脱机数据结合的优越性，突出了在线 RL 和脱机 RL 之间的区别。