Offline reinforcement learning (RL) have received rising interest due to its
appealing data efficiency. The present study addresses behavior estimation, a
task that lays the foundation of many offline RL algorithms. Behavior
estimation aims at estimating the policy with which training data are
generated. In particular, this work considers a scenario where the data are
collected from multiple sources. In this case, neglecting data heterogeneity,
existing approaches for behavior estimation suffers from behavior
misspecification. To overcome this drawback, the present study proposes a
latent variable model to infer a set of policies from data, which allows an
agent to use as behavior policy the policy that best describes a particular
trajectory. This model provides with a agent fine-grained characterization for
multi-source data and helps it overcome behavior misspecification. This work
also proposes a learning algorithm for this model and illustrates its practical
usage via extending an existing offline RL algorithm. Lastly, with extensive
evaluation this work confirms the existence of behavior misspecification and
the efficacy of the proposed model.

该研究提出了一种基于潜变量模型对离线强化学习中多源数据的行为估计进行精细化建模的方法，以解决现有方法在忽略数据异质性时容易出现行为估计错误的问题，并且在现有离线 RL 算法上的实验表明该方法的有效性。