Offline reinforcement learning proposes to learn policies from large
collected datasets without interacting with the physical environment. These
algorithms have made it possible to learn useful skills from data that can then
be deployed in the environment in real-world settings where interactions may be
costly or dangerous, such as autonomous driving or factories. However, current
algorithms overfit to the dataset they are trained on and exhibit poor
out-of-distribution generalization to the environment when deployed. In this
paper, we study the effectiveness of performing data augmentations on the state
space, and study 7 different augmentation schemes and how they behave with
existing offline RL algorithms. We then combine the best data performing
augmentation scheme with a state-of-the-art Q-learning technique, and improve
the function approximation of the Q-networks by smoothening out the learned
state-action space. We experimentally show that using this Surprisingly Simple
Self-Supervision technique in RL (S4RL), we significantly improve over the
current state-of-the-art algorithms on offline robot learning environments such
as MetaWorld [1] and RoboSuite [2,3], and benchmark datasets such as D4RL [4].

本文旨在通过对状态空间进行数据扩充来提高线下强化学习算法的泛化性能，并结合最先进的 Q-learning 技术，通过平滑学习到的状态 - 动作空间，实现在 MetaWorld 和 RoboSuite 等机器人学习环境以及 D4RL 数据集上显著的提升。