In recent years, deep Reinforcement Learning (RL) has been successful in
various combinatorial search domains, such as two-player games and scientific
discovery. However, directly applying deep RL in planning domains is still
challenging. One major difficulty is that without a human-crafted heuristic
function, reward signals remain zero unless the learning framework discovers
any solution plan. Search space becomes \emph{exponentially larger} as the
minimum length of plans grows, which is a serious limitation for planning
instances with a minimum plan length of hundreds to thousands of steps.
Previous learning frameworks that augment graph search with deep neural
networks and extra generated subgoals have achieved success in various
challenging planning domains. However, generating useful subgoals requires
extensive domain knowledge. We propose a domain-independent method that
augments graph search with graph value iteration to solve hard planning
instances that are out of reach for domain-specialized solvers. In particular,
instead of receiving learning signals only from discovered plans, our approach
also learns from failed search attempts where no goal state has been reached.
The graph value iteration component can exploit the graph structure of local
search space and provide more informative learning signals. We also show how we
use a curriculum strategy to smooth the learning process and perform a full
analysis of how graph value iteration scales and enables learning.

该论文提出了一种基于图值迭代的领域无关方法，通过利用局部搜索空间的图结构提供更多的信息学习信号，实现了解决规划任务的目标状态，以及通过一种课程策略来平滑学习过程。

图形价值迭代

Graph Value Iteration

Deep learning methods typically require vast amounts of training data to
reach their full potential. While some publicly available datasets exists,
domain specific data always needs to be collected and manually labeled, an
expensive, time consuming and error prone process. Training with synthetic data
is therefore very lucrative, as dataset creation and labeling comes for free.
We propose a novel method for creating purely synthetic training data for
object detection. We leverage a large dataset of 3D background models and
densely render them using full domain randomization. This yields background
images with realistic shapes and texture on top of which we render the objects
of interest. During training, the data generation process follows a curriculum
strategy guaranteeing that all foreground models are presented to the network
equally under all possible poses and conditions with increasing complexity. As
a result, we entirely control the underlying statistics and we create optimal
training samples at every stage of training. Using a set of 64 retail objects,
we demonstrate that our simple approach enables the training of detectors that
outperform models trained with real data on a challenging evaluation dataset.

本文提出了一种全新的方法来创造纯合成训练数据，通过利用大量的 3D 背景模型和完全领域随机化的方法得到逼真的背景图像，并使用 “课程策略” 来生成所有前景模型，从而得到最佳的训练样本。使用 64 个商品对象进行实验，表明我们的简单方法可以训练出优于实际数据的检测器。