Value approximation using deep neural networks is at the heart of off-policy
deep reinforcement learning, and is often the primary module that provides
learning signals to the rest of the algorithm. While multi-layer perceptron
networks are universal function approximators, recent works in neural kernel
regression suggest the presence of a spectral bias, where fitting
high-frequency components of the value function requires exponentially more
gradient update steps than the low-frequency ones. In this work, we re-examine
off-policy reinforcement learning through the lens of kernel regression and
propose to overcome such bias via a composite neural tangent kernel. With just
a single line-change, our approach, the Fourier feature networks (FFN) produce
state-of-the-art performance on challenging continuous control domains with
only a fraction of the compute. Faster convergence and better off-policy
stability also make it possible to remove the target network without suffering
catastrophic divergences, which further reduces TD}(0)'s estimation bias on a
few tasks.

本文探讨了如何通过使用复合神经切向核的傅里叶特征网络来克服多层感知器和神经核回归中存在的高频率成分拟合所需的步骤数指数级增加的问题，以提高深度增强学习的效率和稳定性，并取得了令人瞩目的实验结果。