Recent works in Task and Motion Planning (TAMP) show that training control
policies on language-supervised robot trajectories with quality labeled data
markedly improves agent task success rates. However, the scarcity of such data
presents a significant hurdle to extending these methods to general use cases.
To address this concern, we present an automated framework to decompose
trajectory data into temporally bounded and natural language-based descriptive
sub-tasks by leveraging recent prompting strategies for Foundation Models (FMs)
including both Large Language Models (LLMs) and Vision Language Models (VLMs).
Our framework provides both time-based and language-based descriptions for
lower-level sub-tasks that comprise full trajectories. To rigorously evaluate
the quality of our automatic labeling framework, we contribute an algorithm
SIMILARITY to produce two novel metrics, temporal similarity and semantic
similarity. The metrics measure the temporal alignment and semantic fidelity of
language descriptions between two sub-task decompositions, namely an FM
sub-task decomposition prediction and a ground-truth sub-task decomposition. We
present scores for temporal similarity and semantic similarity above 90%,
compared to 30% of a randomized baseline, for multiple robotic environments,
demonstrating the effectiveness of our proposed framework. Our results enable
building diverse, large-scale, language-supervised datasets for improved
robotic TAMP.

最近的任务与动作规划研究表明，使用质量标记数据对语言监督的机器人轨迹进行控制策略训练显著提高了智能体任务成功率。然而，这类数据稀缺性在推广这些方法到一般使用案例时存在重大障碍。为了解决这个问题，我们提出了一个自动化框架，通过借鉴 Foundation Models (FMs) 的最新提示策略，包括大型语言模型（LLMs）和视觉语言模型（VLMs），将轨迹数据分解为基于时间限制和自然语言的描述性子任务。我们的框架提供了由时间和语言描述的较低级别子任务组成的全面轨迹描述。为了严格评估我们的自动标记框架的质量，我们提供了一种算法 SIMILARITY 来生成两个新度量标准，即时间相似度和语义相似度。这些度量标准用于衡量两个子任务分解之间的时间对齐和语义逼真度，即 FM 子任务分解预测和基准子任务分解之间的对比。我们的结果显示时间相似度和语义相似度都高达 90% 以上，而随机基线只有 30%，在多个机器人环境中证明了我们提出的框架的有效性。我们的研究结果为构建多样化、大规模的语言监督数据集以改进机器人的任务和动作规划提供了可能。