Partially observable Markov decision processes (POMDP) are a useful model for
decision-making under partial observability and stochastic actions. Partially
Observable Monte-Carlo Planning is an online algorithm for deciding on the next
action to perform, using a Monte-Carlo tree search approach, based on the UCT
(UCB applied to trees) algorithm for fully observable Markov-decision
processes. POMCP develops an action-observation tree, and at the leaves, uses a
rollout policy to provide a value estimate for the leaf. As such, POMCP is
highly dependent on the rollout policy to compute good estimates, and hence
identify good actions. Thus, many practitioners who use POMCP are required to
create strong, domain-specific heuristics.
In this paper, we model POMDPs as stochastic contingent planning problems.
This allows us to leverage domain-independent heuristics that were developed in
the planning community. We suggest two heuristics, the first is based on the
well-known h_add heuristic from classical planning, and the second is computed
in belief space, taking the value of information into account.

部分可观察马尔可夫决策过程 (POMDP) 是在部分可观察性和随机行为下进行决策的有用模型。本文将 POMDP 建模为随机条件规划问题，并提出了两种领域独立的启发式算法，一种基于经典规划中著名的 h_add 启发式算法，另一种在信念空间中计算并考虑信息价值。

在线随机情况规划的展开策略

Rollout Heuristics for Online Stochastic Contingent Planning

Although large language models demonstrate remarkable question-answering
performances, revealing the intermediate reasoning steps that the models
faithfully follow remains challenging. In this paper, we propose FAME (FAithful
question answering with MontE-carlo planning) to answer questions based on
faithful reasoning steps. The reasoning steps are organized as a structured
entailment tree, which shows how premises are used to produce intermediate
conclusions that can prove the correctness of the answer. We formulate the task
as a discrete decision-making problem and solve it through the interaction of a
reasoning environment and a controller. The environment is modular and contains
several basic task-oriented modules, while the controller proposes actions to
assemble the modules. Since the search space could be large, we introduce a
Monte-Carlo planning algorithm to do a look-ahead search and select actions
that will eventually lead to high-quality steps. FAME achieves state-of-the-art
performance on the standard benchmark. It can produce valid and faithful
reasoning steps compared with large language models with a much smaller model
size.

本论文提出 FAithful question answering with MontE-carlo planning （FAME）方法，以结构化蕴含树方式展示中间推理步骤，包含多个基础任务模块并通过蒙特卡罗规划算法作为控制器提出行动，旨在回答问题并与大型语言模型相比，使用更小的模型大小实现了最先进的性能。

蒙特卡罗规划的忠实问答

Faithful Question Answering with Monte-Carlo Planning

Monte-Carlo planning, as exemplified by Monte-Carlo Tree Search (MCTS), has
demonstrated remarkable performance in applications with finite spaces. In this
paper, we consider Monte-Carlo planning in an environment with continuous
state-action spaces, a much less understood problem with important applications
in control and robotics. We introduce POLY-HOOT, an algorithm that augments
MCTS with a continuous armed bandit strategy named Hierarchical Optimistic
Optimization (HOO) (Bubeck et al., 2011). Specifically, we enhance HOO by using
an appropriate polynomial, rather than logarithmic, bonus term in the upper
confidence bounds. Such a polynomial bonus is motivated by its empirical
successes in AlphaGo Zero (Silver et al., 2017b), as well as its significant
role in achieving theoretical guarantees of finite space MCTS (Shah et al.,
2019). We investigate, for the first time, the regret of the enhanced HOO
algorithm in non-stationary bandit problems. Using this result as a building
block, we establish non-asymptotic convergence guarantees for POLY-HOOT: the
value estimate converges to an arbitrarily small neighborhood of the optimal
value function at a polynomial rate. We further provide experimental results
that corroborate our theoretical findings.

本文提出了一种算法 POLY-HOOT，将连续武装匪徒策略与 Monte-Carlo Tree Search（MCTS）相结合，使用多项式奖励项来增强 HOO 算法，并分析其在非静态匪徒问题中的后悔率和收敛性。