Existing theoretical studies on offline reinforcement learning (RL) mostly
consider a dataset sampled directly from the target task. In practice, however,
data often come from several heterogeneous but related sources. Motivated by
this gap, this work aims at rigorously understanding offline RL with multiple
datasets that are collected from randomly perturbed versions of the target task
instead of from itself. An information-theoretic lower bound is derived, which
reveals a necessary requirement on the number of involved sources in addition
to that on the number of data samples. Then, a novel HetPEVI algorithm is
proposed, which simultaneously considers the sample uncertainties from a finite
number of data samples per data source and the source uncertainties due to a
finite number of available data sources. Theoretical analyses demonstrate that
HetPEVI can solve the target task as long as the data sources collectively
provide a good data coverage. Moreover, HetPEVI is demonstrated to be optimal
up to a polynomial factor of the horizon length. Finally, the study is extended
to offline Markov games and offline robust RL, which demonstrates the
generality of the proposed designs and theoretical analyses.

本文研究了使用多个数据集进行离线强化学习的问题，提出了一种考虑样本和数据源不确定性的算法，并在离线马尔科夫博弈和离线鲁棒强化学习方面进行了推广。