Recent advancements in multimodal Human-Robot Interaction (HRI) datasets have
highlighted the fusion of speech and gesture, expanding robots' capabilities to
absorb explicit and implicit HRI insights. However, existing speech-gesture HRI
datasets often focus on elementary tasks, like object pointing and pushing,
revealing limitations in scaling to intricate domains and prioritizing human
command data over robot behavior records. To bridge these gaps, we introduce
NatSGD, a multimodal HRI dataset encompassing human commands through speech and
gestures that are natural, synchronized with robot behavior demonstrations.
NatSGD serves as a foundational resource at the intersection of machine
learning and HRI research, and we demonstrate its effectiveness in training
robots to understand tasks through multimodal human commands, emphasizing the
significance of jointly considering speech and gestures. We have released our
dataset, simulator, and code to facilitate future research in human-robot
interaction system learning; access these resources at
this https URL

NatSGD 是一个多模式人机交互数据集，在机器学习和人机交互研究交叉领域起到基础性的作用；我们通过训练机器人理解多模态人类指令的任务来展示其效果，并强调了同时考虑语音和手势的重要性。