Offline optimization paradigms such as offline Reinforcement Learning (RL) or
Imitation Learning (IL) allow policy search algorithms to make use of offline
data, but require careful incorporation of uncertainty in order to circumvent
the challenges of distribution shift. Gradient-based policy search methods are
a promising direction due to their effectiveness in high dimensions; however,
we require a more careful consideration of how these methods interplay with
uncertainty estimation. We claim that in order for an uncertainty metric to be
amenable for gradient-based optimization, it must be (i) stably convergent to
data when uncertainty is minimized with gradients, and (ii) not prone to
underestimation of true uncertainty. We investigate smoothed distance to data
as a metric, and show that it not only stably converges to data, but also
allows us to analyze model bias with Lipschitz constants. Moreover, we
establish an equivalence between smoothed distance to data and data likelihood,
which allows us to use score-matching techniques to learn gradients of distance
to data. Importantly, we show that offline model-based policy search problems
that maximize data likelihood do not require values of likelihood; but rather
only the gradient of the log likelihood (the score function). Using this
insight, we propose Score-Guided Planning (SGP), a planning algorithm for
offline RL that utilizes score-matching to enable first-order planning in
high-dimensional problems, where zeroth-order methods were unable to scale, and
ensembles were unable to overcome local minima. Website:
this https URL

该研究探究了使用离线 RL 和 IL 进行策略搜索算法的离线优化范式，并要求更加小心地考虑这些方法是如何与不确定性评估相互作用的。同时，该研究提出了一种名为 Score-Guided Planning（SGP）的规划算法，它利用分数匹配来实现高维问题中的一阶规划。