Reinforcement learning (RL) agents can leverage batches of previously
collected data to extract a reasonable control policy. An emerging issue in
this offline RL setting, however, is that the bootstrapping update underlying
many of our methods suffers from insufficient action-coverage: standard max
operator may select a maximal action that has not been seen in the dataset.
Bootstrapping from these inaccurate values can lead to overestimation and even
divergence. There are a growing number of methods that attempt to approximate
an \emph{in-sample} max, that only uses actions well-covered by the dataset. We
highlight a simple fact: it is more straightforward to approximate an in-sample
\emph{softmax} using only actions in the dataset. We show that policy iteration
based on the in-sample softmax converges, and that for decreasing temperatures
it approaches the in-sample max. We derive an In-Sample Actor-Critic (AC),
using this in-sample softmax, and show that it is consistently better or
comparable to existing offline RL methods, and is also well-suited to
fine-tuning.

研究表明，在强化学习中，使用数据集中的动作来逼近 softmax 比逼近 max 更为简单，并提出基于此的在线学习算法 In-Sample Actor-Critic，该算法表现良好且适合微调。