Controlling agents remotely with deep reinforcement learning~(DRL) in the
real world is yet to come. One crucial stepping stone is to devise RL
algorithms that are robust in the face of dropped information from corrupted
communication or malfunctioning sensors. Typical RL methods usually require
considerable online interaction data that are costly and unsafe to collect in
the real world. Furthermore, when applying to the frame dropping scenarios,
they perform unsatisfactorily even with moderate drop rates. To address these
issues, we propose Decision Transformer under Random Frame Dropping~(DeFog), an
offline RL algorithm that enables agents to act robustly in frame dropping
scenarios without online interaction. DeFog first randomly masks out data in
the offline datasets and explicitly adds the time span of frame dropping as
inputs. After that, a finetuning stage on the same offline dataset with a
higher mask rate would further boost the performance. Empirical results show
that DeFog outperforms strong baselines under severe frame drop rates like
90\%, while maintaining similar returns under non-frame-dropping conditions in
the regular MuJoCo control benchmarks and the Atari environments. Our approach
offers a robust and deployable solution for controlling agents in real-world
environments with limited or unreliable data.

本文提出了一种名为 “基于随机帧丢失的决策变换器” 的离线强化学习算法，可以使代理在帧丢失场景下稳健地行动，同时不需要在线交互数据，该算法通过随机掩蔽离线数据并显式地添加帧丢失的时间跨度作为输入，使用相同的离线数据集进行微调，可以在严重的帧丢失率下优于强基线，同时在常规的 MuJoCo 控制基准和 Atari 环境中具有相似的回报。该方法提供了一个稳健和可部署的解决方案，以控制在具有有限或不可靠数据的实际环境中的代理。