Online tree-based search algorithms iteratively simulate trajectories and
update action-values for a set of states stored in a tree structure. It works
reasonably well in practice but fails to effectively utilise the information
gathered from similar states. Depending upon the smoothness of the action-value
function, one approach to overcoming this issue is through online learning,
where information is interpolated among similar states; Policy Gradient Search
provides a practical algorithm to achieve this. However, Policy Gradient Search
lacks an explicit exploration mechanism, which is a key feature of tree-based
online search algorithms. In this paper, we propose an efficient and effective
online search algorithm called Exploratory Policy Gradient Search (ExPoSe),
which leverages information sharing among states by updating the search policy
parameters directly, while incorporating a well-defined exploration mechanism
during the online search process. We evaluate ExPoSe on a range of
decision-making problems, including Atari games, Sokoban, and Hamiltonian cycle
search in sparse graphs. The results demonstrate that ExPoSe consistently
outperforms other popular online search algorithms across all domains. The
ExPoSe source code is available at
\textit{https://github.com/dixantmittal/ExPoSe}.

本论文提出一种高效有效的在线搜索算法 —— 探索性策略梯度搜索（ExPoSe），结合明确定义的探索机制，并在在线搜索过程中直接更新搜索策略参数，利用状态之间的信息共享，评估其在各种决策制定问题中的性能表现（包括 Atari 游戏、Sokoban 和稀疏图中的 Hamilton 圈搜索），结果表明 ExPoSe 在所有领域中始终优于其他流行的在线搜索算法。