While deep reinforcement learning (RL) has been demonstrated effective in
solving complex control tasks, sample efficiency remains a key challenge due to
the large amounts of data required for remarkable performance. Existing
research explores the application of representation learning for data-efficient
RL, e.g., learning predictive representations by predicting long-term future
states. However, many existing methods do not fully exploit the structural
information inherent in sequential state signals, which can potentially improve
the quality of long-term decision-making but is difficult to discern in the
time domain. To tackle this problem, we propose State Sequences Prediction via
Fourier Transform (SPF), a novel method that exploits the frequency domain of
state sequences to extract the underlying patterns in time series data for
learning expressive representations efficiently. Specifically, we theoretically
analyze the existence of structural information in state sequences, which is
closely related to policy performance and signal regularity, and then propose
to predict the Fourier transform of infinite-step future state sequences to
extract such information. One of the appealing features of SPF is that it is
simple to implement while not requiring storage of infinite-step future states
as prediction targets. Experiments demonstrate that the proposed method
outperforms several state-of-the-art algorithms in terms of both sample
efficiency and performance.

通过 Fourier 变换，我们提出了 State Sequences Prediction via Fourier Transform (SPF) 方法，用于在时间序列数据中提取结构信息，以实现数据高效的 RL，实验证明该方法在样本效率和性能方面优于多种现有算法。