Recent deep learning approaches have achieved impressive performance on
visual sound separation tasks. However, these approaches are mostly built on
appearance and optical flow like motion feature representations, which exhibit
limited abilities to find the correlations between audio signals and visual
points, especially when separating multiple instruments of the same types, such
as multiple violins in a scene. To address this, we propose "Music Gesture," a
keypoint-based structured representation to explicitly model the body and
finger movements of musicians when they perform music. We first adopt a
context-aware graph network to integrate visual semantic context with body
dynamics, and then apply an audio-visual fusion model to associate body
movements with the corresponding audio signals. Experimental results on three
music performance datasets show: 1) strong improvements upon benchmark metrics
for hetero-musical separation tasks (i.e. different instruments); 2) new
ability for effective homo-musical separation for piano, flute, and trumpet
duets, which to our best knowledge has never been achieved with alternative
methods. Project page: this http URL

本研究基于 Music Gesture 模型，提出了一种基于关键点的结构化表示法来建模音乐家演奏的身体和手指运动，并在视觉和音频结合分离任务中实现了较强的音频分离效果。

音乐手势用于视听分离

Music Gesture for Visual Sound Separation

Sounds provide rich semantics, complementary to visual data, for many tasks.
However, in practice, sounds from multiple sources are often mixed together. In
this paper we propose a novel framework, referred to as MinusPlus Network
(MP-Net), for the task of visual sound separation. MP-Net separates sounds
recursively in the order of average energy, removing the separated sound from
the mixture at the end of each prediction, until the mixture becomes empty or
contains only noise. In this way, MP-Net could be applied to sound mixtures
with arbitrary numbers and types of sounds. Moreover, while MP-Net keeps
removing sounds with large energy from the mixture, sounds with small energy
could emerge and become clearer, so that the separation is more accurate.
Compared to previous methods, MP-Net obtains state-of-the-art results on two
large scale datasets, across mixtures with different types and numbers of
sounds.

本篇论文提出了一种名为 MinusPlus Network (MP-Net) 的新型框架，用于视听分离任务。MP-Net 按平均能量的顺序递归地分离声音，并将分离后的声音从混合物中移除，直到混合物为空或仅包含噪声。通过这种方式，MP-Net 可以应用于具有任意数量和类型声音的混音中，并且相较于以前的方法取得了最先进的结果。