We hypothesize dance as a motion that forms a visual rhythm from music, where
the visual rhythm can be perceived from an optical flow. If an agent can
recognize the relationship between visual rhythm and music, it will be able to
dance by generating a motion to create a visual rhythm that matches the music.
Based on this, we propose a framework for any kind of non-humanoid agents to
learn how to dance from human videos. Our framework works in two processes: (1)
training a reward model which perceives the relationship between optical flow
(visual rhythm) and music from human dance videos, (2) training the
non-humanoid dancer based on that reward model, and reinforcement learning. Our
reward model consists of two feature encoders for optical flow and music. They
are trained based on contrastive learning which makes the higher similarity
between concurrent optical flow and music features. With this reward model, the
agent learns dancing by getting a higher reward when its action creates an
optical flow whose feature has a higher similarity with the given music
feature. Experiment results show that generated dance motion can align with the
music beat properly, and user study result indicates that our framework is more
preferred by humans compared to the baselines. To the best of our knowledge,
our work of non-humanoid agents which learn dance from human videos is
unprecedented. An example video can be found at this https URL

我们提出了一个框架，使非人形代理能够从人类视频中学习跳舞，通过训练一个奖励模型，该模型可以感知光流（视觉节奏）和音乐之间的关系，并通过生成与音乐匹配的视觉节奏的动作来学习跳舞。