Recent advancements in multimodal Human-Robot Interaction (HRI) datasets have
highlighted the fusion of speech and gesture, expanding robots' capabilities to
absorb explicit and implicit HRI insights. However, existing speech-gesture HRI
datasets often focus on elementary tasks, like object pointing and pushing,
revealing limitations in scaling to intricate domains and prioritizing human
command data over robot behavior records. To bridge these gaps, we introduce
NatSGD, a multimodal HRI dataset encompassing human commands through speech and
gestures that are natural, synchronized with robot behavior demonstrations.
NatSGD serves as a foundational resource at the intersection of machine
learning and HRI research, and we demonstrate its effectiveness in training
robots to understand tasks through multimodal human commands, emphasizing the
significance of jointly considering speech and gestures. We have released our
dataset, simulator, and code to facilitate future research in human-robot
interaction system learning; access these resources at
this https URL

NatSGD 是一个多模式人机交互数据集，在机器学习和人机交互研究交叉领域起到基础性的作用；我们通过训练机器人理解多模态人类指令的任务来展示其效果，并强调了同时考虑语音和手势的重要性。

NatSGD: 自然人机交互中机器人学习的包含语音、手势和演示的数据集

NatSGD: A Dataset with Speech, Gestures, and Demonstrations for Robot  Learning in Natural Human-Robot Interaction

Gesture as language of non-verbal communication has been theoretically
established since the 17th century. However, its relevance for the visual arts
has been expressed only sporadically. This may be primarily due to the sheer
overwhelming amount of data that traditionally had to be processed by hand.
With the steady progress of digitization, though, a growing number of
historical artifacts have been indexed and made available to the public,
creating a need for automatic retrieval of art-historical motifs with similar
body constellations or poses. Since the domain of art differs significantly
from existing real-world data sets for human pose estimation due to its style
variance, this presents new challenges. In this paper, we propose a novel
approach to estimate human poses in art-historical images. In contrast to
previous work that attempts to bridge the domain gap with pre-trained models or
through style transfer, we suggest semi-supervised learning for both object and
keypoint detection. Furthermore, we introduce a novel domain-specific art data
set that includes both bounding box and keypoint annotations of human figures.
Our approach achieves significantly better results than methods that use
pre-trained models or style transfer.

本文提出了一种在艺术历史图像中估计人类姿势的新方法，通过半监督学习对目标和关键点检测进行了改进，并且引入了一个包括人体姿势标注的新领域特定艺术数据集，相对于使用预训练模型或风格转移的方法，达到了显著更好的结果。