In Extreme Q-learning (XQL), Gumbel Regression is performed with an assumed
Gumbel distribution for the error distribution. This allows learning of the
value function without sampling out-of-distribution actions and has shown
excellent performance mainly in Offline RL. However, issues remained, including
the exponential term in the loss function causing instability and the potential
for an error distribution diverging from the Gumbel distribution. Therefore, we
propose Maclaurin Expanded Extreme Q-learning to enhance stability. In this
method, applying Maclaurin expansion to the loss function in XQL enhances
stability against large errors. It also allows adjusting the error distribution
assumption from normal to Gumbel based on the expansion order. Our method
significantly stabilizes learning in Online RL tasks from DM Control, where XQL
was previously unstable. Additionally, it improves performance in several
Offline RL tasks from D4RL, where XQL already showed excellent results.

通过应用 Maclaurin 展开来改进 Extreme Q-learning 方法，提高了稳定性，并允许根据展开阶数调整误差分布假设，从而显著稳定了在线强化学习任务，同时提高了离线强化学习任务的性能。