In decision-making problems with limited training data, policy functions
approximated using deep neural networks often exhibit suboptimal performance.
An alternative approach involves learning a world model from the limited data
and determining actions through online search. However, the performance is
adversely affected by compounding errors arising from inaccuracies in the
learnt world model. While methods like TreeQN have attempted to address these
inaccuracies by incorporating algorithmic structural biases into their
architectures, the biases they introduce are often weak and insufficient for
complex decision-making tasks. In this work, we introduce Differentiable Tree
Search (DTS), a novel neural network architecture that significantly
strengthens the inductive bias by embedding the algorithmic structure of a
best-first online search algorithm. DTS employs a learnt world model to conduct
a fully differentiable online search in latent state space. The world model is
jointly optimised with the search algorithm, enabling the learning of a robust
world model and mitigating the effect of model inaccuracies. We address
potential Q-function discontinuities arising from naive incorporation of
best-first search by adopting a stochastic tree expansion policy, formulating
search tree expansion as a decision-making task, and introducing an effective
variance reduction technique for the gradient computation. We evaluate DTS in
an offline-RL setting with a limited training data scenario on Procgen games
and grid navigation task, and demonstrate that DTS outperforms popular
model-free and model-based baselines.

在决策问题中，由于训练数据有限，使用深度神经网络逼近的策略函数常常表现出亚优性。作者引入了一种名为 Differentiable Tree Search (DTS) 的新颖神经网络架构，通过嵌入最佳优先在线搜索算法的算法结构，显著增强了归纳偏置，从而解决了模型精度不准确导致的性能问题。DTS 采用一个学习得到的世界模型在潜在状态空间中进行完全可微的在线搜索，并通过优化搜索算法和世界模型来学习到鲁棒的世界模型，减轻模型不准确性的影响。通过在有限训练数据情境下的 Procgen 游戏和网格导航任务中离线强化学习实验，证明 DTS 优于常见的无模型和有模型的基准算法。