Sounds carry an abundance of information about activities and events in our
everyday environment, such as traffic noise, road works, music, or people
talking. Recent machine learning methods, such as convolutional neural networks
(CNNs), have been shown to be able to automatically recognize sound activities,
a task known as audio tagging. One such method, pre-trained audio neural
networks (PANNs), provides a neural network which has been pre-trained on over
500 sound classes from the publicly available AudioSet dataset, and can be used
as a baseline or starting point for other tasks. However, the existing PANNs
model has a high computational complexity and large storage requirement. This
could limit the potential for deploying PANNs on resource-constrained devices,
such as on-the-edge sound sensors, and could lead to high energy consumption if
many such devices were deployed. In this paper, we reduce the computational
complexity and memory requirement of the PANNs model by taking a pruning
approach to eliminate redundant parameters from the PANNs model. The resulting
Efficient PANNs (E-PANNs) model, which requires 36\% less computations and 70\%
less memory, also slightly improves the sound recognition (audio tagging)
performance. The code for the E-PANNs model has been released under an open
source license.

通过裁剪方法减少预训练音频神经网络 (PANNs) 的冗余参数，降低了计算复杂度和存储需求，提高了声音识别表现，从而可在资源受限的设备上进行部署。

E-PANNs: 使用高效的预训练音频神经网络进行声音识别

E-PANNs: Sound Recognition Using Efficient Pre-trained Audio Neural  Networks

Recognizing sounds is a key aspect of computational audio scene analysis and
machine perception. In this paper, we advocate that sound recognition is
inherently a multi-modal audiovisual task in that it is easier to differentiate
sounds using both the audio and visual modalities as opposed to one or the
other. We present an audiovisual fusion model that learns to recognize sounds
from weakly labeled video recordings. The proposed fusion model utilizes an
attention mechanism to dynamically combine the outputs of the individual audio
and visual models. Experiments on the large scale sound events dataset,
AudioSet, demonstrate the efficacy of the proposed model, which outperforms the
single-modal models, and state-of-the-art fusion and multi-modal models. We
achieve a mean Average Precision (mAP) of 46.16 on Audioset, outperforming
prior state of the art by approximately +4.35 mAP (relative: 10.4%).

本文提出了一种音频视觉融合模型，该模型利用注意机制动态地结合单独的音频和视觉模型的输出来识别声音，实验证明该模型在音频场景分析和机器感知上比单模和多模融合模型具有更好的效果。

利用弱标签数据进行大规模音频视觉学习

Large Scale Audiovisual Learning of Sounds with Weakly Labeled Data

Humans do not acquire perceptual abilities in the way we train machines.
While machine learning algorithms typically operate on large collections of
randomly-chosen, explicitly-labeled examples, human acquisition relies more
heavily on multimodal unsupervised learning (as infants) and active learning
(as children). With this motivation, we present a learning framework for sound
representation and recognition that combines (i) a self-supervised objective
based on a general notion of unimodal and cross-modal coincidence, (ii) a
clustering objective that reflects our need to impose categorical structure on
our experiences, and (iii) a cluster-based active learning procedure that
solicits targeted weak supervision to consolidate categories into relevant
semantic classes. By training a combined sound
embedding/clustering/classification network according to these criteria, we
achieve a new state-of-the-art unsupervised audio representation and
demonstrate up to a 20-fold reduction in the number of labels required to reach
a desired classification performance.

提出一种结合机器学习中自我监督、聚类目标和主动式学习的学习框架，以改善音频表征和识别，实现新的最先进的无监督音频表征并证明了它的有效性。

巧合、分类和巩固：学习在最小监督下识别声音

Coincidence, Categorization, and Consolidation: Learning to Recognize  Sounds with Minimal Supervision

Deep learning methods have achieved high performance in sound recognition
tasks. Deciding how to feed the training data is important for further
performance improvement. We propose a novel learning method for deep sound
recognition: Between-Class learning (BC learning). Our strategy is to learn a
discriminative feature space by recognizing the between-class sounds as
between-class sounds. We generate between-class sounds by mixing two sounds
belonging to different classes with a random ratio. We then input the mixed
sound to the model and train the model to output the mixing ratio. The
advantages of BC learning are not limited only to the increase in variation of
the training data; BC learning leads to an enlargement of Fisher's criterion in
the feature space and a regularization of the positional relationship among the
feature distributions of the classes. The experimental results show that BC
learning improves the performance on various sound recognition networks,
datasets, and data augmentation schemes, in which BC learning proves to be
always beneficial. Furthermore, we construct a new deep sound recognition
network (EnvNet-v2) and train it with BC learning. As a result, we achieved a
performance surpasses the human level.

该研究提出了一种名为 BC learning 的新型深度学习方法，通过识别类间声音来学习区分性特征空间，该方法不仅仅可以增加训练数据的多样性，还可以增加 Fisher 的判据，规范类之间特征分布的关系。该方法可以在各种声音识别网络、数据集和数据增强方案中提高性能，并通过构建深度声音识别网络 EnvNet-v2 并使用 BC learning 方法，实现了超越人类的性能。