In visual Reinforcement Learning (RL), upstream representation learning
largely determines the effect of downstream policy learning. Employing
auxiliary tasks allows the agent to enhance visual representation in a targeted
manner, thereby improving the sample efficiency and performance of downstream
RL. Prior advanced auxiliary tasks all focus on how to extract as much
information as possible from limited experience (including observations,
actions, and rewards) through their different auxiliary objectives, whereas in
this article, we first start from another perspective: auxiliary training data.
We try to improve auxiliary representation learning for RL by enriching
auxiliary training data, proposing \textbf{L}earning \textbf{F}uture
representation with \textbf{S}ynthetic observations \textbf{(LFS)}, a novel
self-supervised RL approach. Specifically, we propose a training-free method to
synthesize observations that may contain future information, as well as a data
selection approach to eliminate unqualified synthetic noise. The remaining
synthetic observations and real observations then serve as the auxiliary data
to achieve a clustering-based temporal association task for representation
learning. LFS allows the agent to access and learn observations that have not
yet appeared in advance, so as to quickly understand and exploit them when they
occur later. In addition, LFS does not rely on rewards or actions, which means
it has a wider scope of application (e.g., learning from video) than recent
advanced auxiliary tasks. Extensive experiments demonstrate that our LFS
exhibits state-of-the-art RL sample efficiency on challenging continuous
control and enables advanced visual pre-training based on action-free video
demonstrations.

通过丰富辅助训练数据，提出了一种无需训练的方法来合成可能包含未来信息的观察，从而改进了强化学习中的辅助表示学习，并展示了其在连续控制和基于无动作视频演示的视觉预训练中的先进性能。

使用合成观测学习未来表示以提高效率的强化学习

Learning Future Representation with Synthetic Observations for  Sample-efficient Reinforcement Learning

Interpreting the observations of exoplanet atmospheres to constrain physical
and chemical properties is typically done using Bayesian retrieval techniques.
Because these methods require many model computations, a compromise is made
between model complexity and run time. Reaching this compromise leads to the
simplification of many physical and chemical processes (e.g. parameterised
temperature structure). Here we implement and test sequential neural posterior
estimation (SNPE), a machine learning inference algorithm, for exoplanet
atmospheric retrievals. The goal is to speed up retrievals so they can be run
with more computationally expensive atmospheric models, such as those computing
the temperature structure using radiative transfer. We generate 100 synthetic
observations using ARCiS (ARtful Modeling Code for exoplanet Science, an
atmospheric modelling code with the flexibility to compute models in varying
degrees of complexity) and perform retrievals on them to test the faithfulness
of the SNPE posteriors. The faithfulness quantifies whether the posteriors
contain the ground truth as often as we expect. We also generate a synthetic
observation of a cool brown dwarf using the self-consistent capabilities of
ARCiS and run a retrieval with self-consistent models to showcase the
possibilities that SNPE opens. We find that SNPE provides faithful posteriors
and is therefore a reliable tool for exoplanet atmospheric retrievals. We are
able to run a self-consistent retrieval of a synthetic brown dwarf spectrum
using only 50,000 forward model evaluations. We find that SNPE can speed up
retrievals between $\sim2\times$ and $\geq10\times$ depending on the
computational load of the forward model, the dimensionality of the observation,
and the signal-to-noise ratio of the observation. We make the code publicly
available for the community on Github.

利用机器学习推断算法 SNPE，我们能够以更高的计算效率运行可靠的外行星大气解译检索，并加速检索速度，提高计算耗费大气模型的可行性。