In this paper, we investigate how to learn rich and robust feature
representations for audio classification from visual data and acoustic images,
a novel audio data modality. Former models learn audio representations from raw
signals or spectral data acquired by a single microphone, with remarkable
results in classification and retrieval. However, such representations are not
so robust towards variable environmental sound conditions. We tackle this
drawback by exploiting a new multimodal labeled action recognition dataset
acquired by a hybrid audio-visual sensor that provides RGB video, raw audio
signals, and spatialized acoustic data, also known as acoustic images, where
the visual and acoustic images are aligned in space and synchronized in time.
Using this richer information, we train audio deep learning models in a
teacher-student fashion. In particular, we distill knowledge into audio
networks from both visual and acoustic image teachers. Our experiments suggest
that the learned representations are more powerful and have better
generalization capabilities than the features learned from models trained using
just single-microphone audio data.

本研究旨在研究如何从视觉数据和新型的音频数据模式 —— 声学图像中学习丰富和强大的音频分类特征表示，通过利用新的多模态标记行动识别数据集，并针对性地训练音频深度学习模型，从视觉和声学图像方面实现知识蒸馏，以获得比单麦克风声音数据训练模型更强大和更具有一般化能力的特征表示。