We present the Human And Robot Multimodal Observations of Natural Interactive
Collaboration (HARMONIC) data set. This is a large multimodal data set of human
interactions with a robotic arm in a shared autonomy setting designed to
imitate assistive eating. The data set provides human, robot, and environmental
data views of twenty-four different people engaged in an assistive eating task
with a 6 degree-of-freedom (DOF) robot arm. From each participant, we recorded
video of both eyes, egocentric video from a head-mounted camera, joystick
commands, electromyography from the forearm used to operate the joystick, third
person stereo video, and the joint positions of the 6 DOF robot arm. Also
included are several features that come as a direct result of these recordings,
such as eye gaze projected onto the egocentric video, body pose, hand pose, and
facial keypoints. These data streams were collected specifically because they
have been shown to be closely related to human mental states and intention.
This data set could be of interest to researchers studying intention
prediction, human mental state modeling, and shared autonomy. Data streams are
provided in a variety of formats such as video and human-readable CSV and YAML
files.

该研究开发了 HARMONIC 数据集，该数据集包含 24 个不同的人参与 6 自由度机械臂的辅助进食任务的人，机器人和环境的数据视图，这些数据包括眼部视频，头戴式摄像机的自我视频，操纵杆命令，前臂使用的肌电图，第三人称立体视听视频以及机械臂的关节位置，可以帮助研究者推断人类的意图和心理状态。