Semi-gradient Q-learning is applied in many fields, but due to the absence of
an explicit loss function, studying its dynamics and implicit bias in the
parameter space is challenging. This paper introduces the Fokker--Planck
equation and employs partial data obtained through sampling to construct and
visualize the effective loss landscape within a two-dimensional parameter
space. This visualization reveals how the global minima in the loss landscape
can transform into saddle points in the effective loss landscape, as well as
the implicit bias of the semi-gradient method. Additionally, we demonstrate
that saddle points, originating from the global minima in loss landscape, still
exist in the effective loss landscape under high-dimensional parameter spaces
and neural network settings. This paper develop a novel approach for probing
implicit bias in semi-gradient Q-learning.

该论文介绍了在二维参数空间中构建和可视化有效损失景观，揭示了全局最小值如何转化为有效损失景观中的鞍点，以及半梯度方法的隐含偏差。此外，论文还证明了高维参数空间和神经网络设置下，从损失景观中的全局最小值产生的鞍点仍存在于有效损失景观中。该论文开发了一种新的方法来探究半梯度 Q-learning 中的隐含偏差。