Using generative models to synthesize new data has become a de-facto standard
in autonomous driving to address the data scarcity issue. Though existing
approaches are able to boost perception models, we discover that these
approaches fail to improve the performance of planning of end-to-end autonomous
driving models as the generated videos are usually less than 8 frames and the
spatial and temporal inconsistencies are not negligible. To this end, we
propose Delphi, a novel diffusion-based long video generation method with a
shared noise modeling mechanism across the multi-views to increase spatial
consistency, and a feature-aligned module to achieves both precise
controllability and temporal consistency. Our method can generate up to 40
frames of video without loss of consistency which is about 5 times longer
compared with state-of-the-art methods. Instead of randomly generating new
data, we further design a sampling policy to let Delphi generate new data that
are similar to those failure cases to improve the sample efficiency. This is
achieved by building a failure-case driven framework with the help of
pre-trained visual language models. Our extensive experiment demonstrates that
our Delphi generates a higher quality of long videos surpassing previous
state-of-the-art methods. Consequentially, with only generating 4% of the
training dataset size, our framework is able to go beyond perception and
prediction tasks, for the first time to the best of our knowledge, boost the
planning performance of the end-to-end autonomous driving model by a margin of
25%.

使用生成模型合成新数据已成为自动驾驶中解决数据稀缺问题的标准，然而现有方法在提高感知模型方面有效，但我们发现这些方法未能提高端到端自动驾驶模型的规划性能，原因在于生成的视频通常少于 8 帧且空间和时间不一致，为此我们提出了一种基于扩散的长视频生成方法 Delphi，通过跨多视角的共享噪声建模机制提高空间一致性，并引入特征对齐模块实现精确可控性和时间一致性，相比现有方法，我们的方法最多可生成 40 帧的视频而保持一致性，这是现有方法的 5 倍之多，进一步我们通过构建一个失败案例驱动框架，借助预训练视觉语言模型，设计了一种采样策略让 Delphi 生成类似于那些失败案例的新数据，提高了样本效率，广泛的实验证明我们的 Delphi 在驾驶规划中生成更高质量的长视频，首次超越现有最先进的方法，而且仅使用了训练数据集的 4％，我们的框架能够将自动驾驶模型的规划性能提升 25％。

通过可控的长视频生成释放自主驾驶的泛化能力

Unleashing Generalization of End-to-End Autonomous Driving with  Controllable Long Video Generation

A recursive tree planner (RTP) is designed to function as a pure planner
without policies at one extreme and run a pure greedy policy at the other. In
between, the RTP exploits policies to improve planning performance and improve
zero-shot transfer from one class of planning problem to another. Policies are
learned through imitation of the planner. These are then used by the planner to
improve policies in a virtuous cycle. To improve planning performance and
zero-shot transfer, the RTP incorporates previously learned tasks as
generalized actions (GA) at any level of its hierarchy, and can refine those GA
by adding primitive actions at any level too. For search, the RTP uses a
generalized Dijkstra algorithm [Dijkstra 1959] which tries the greedy policy
first and then searches over near-greedy paths and then farther away as
necessary. The RPT can return multiple sub-goals from lower levels as well as
boundary states near obstacles, and can exploit policies with background and
object-number invariance. Policies at all levels of the hierarchy can be
learned simultaneously or in any order or come from outside the framework. The
RTP is tested here on a variety of Box2d [Cato 2022] problems, including the
classic lunar lander [Farama 2022], and on the MuJoCo [Todorov et al 2012]
inverted pendulum.

递归树规划（RTP）同时利用策略和贪心算法来改善规划性能和零样本迁移，使用广义 Dijkstra 算法作为搜索方法，将已学习的任务作为广义动作集合以提高规划性能，并在不同层次上添加原始动作。

从纯计划到纯策略与之间：采用递归树规划方法

Pure Planning to Pure Policies and In Between with a Recursive Tree  Planner

Large Language Models (LLMs) have demonstrated great potential in complex
reasoning tasks, yet they fall short when tackling more sophisticated
challenges, especially when interacting with environments through generating
executable actions. This inadequacy primarily stems from the lack of built-in
action knowledge in language agents, which fails to effectively guide the
planning trajectories during task solving and results in planning
hallucination. To address this issue, we introduce KnowAgent, a novel approach
designed to enhance the planning capabilities of LLMs by incorporating explicit
action knowledge. Specifically, KnowAgent employs an action knowledge base and
a knowledgeable self-learning strategy to constrain the action path during
planning, enabling more reasonable trajectory synthesis, and thereby enhancing
the planning performance of language agents. Experimental results on HotpotQA
and ALFWorld based on various backbone models demonstrate that KnowAgent can
achieve comparable or superior performance to existing baselines. Further
analysis indicates the effectiveness of KnowAgent in terms of planning
hallucinations mitigation. Code is available in
this https URL

通过引入显式的行动知识，KnowAgent 为大型语言模型（LLMs）的规划能力提供了增强，实现了更合理的轨迹合成，从而提高了语言代理的规划性能。