This paper presents a framework to automate the labelling process for
gestures in musical performance videos with a 3D Convolutional Neural Network
(CNN). While this idea was proposed in a previous study, this paper introduces
several novelties: (i) Presents a novel method to overcome the class imbalance
challenge and make learning possible for co-existent gestures by batch
balancing approach and spatial-temporal representations of gestures. (ii)
Performs a detailed study on 7 and 18 categories of gestures generated during
the performance (guitar play) of musical pieces that have been video-recorded.
(iii) Investigates the possibility to use audio features. (iv) Extends the
analysis to multiple videos. The novel methods significantly improve the
performance of gesture identification by 12 %, when compared to the previous
work (51 % in this study over 39 % in previous work). We successfully validate
the proposed methods on 7 super classes (72 %), an ensemble of the 18
gestures/classes, and additional videos (75 %).

该论文介绍了一个自动化标注音乐表演视频中手势的框架，使用 3D 卷积神经网络，通过批处理平衡和空间 - 时间手势表示等方法，成功提高了 12% 的手势识别准确率，对跨多个视频的多个手势 / 类别进行了详细的研究，并探讨了使用音频特征的可能性。