This study defines a new evaluation metric for audio tagging tasks to
overcome the limitation of the conventional mean average precision (mAP)
metric, which treats different kinds of sound as independent classes without
considering their relations. Also, due to the ambiguities in sound labeling,
the labels in the training and evaluation set are not guaranteed to be accurate
and exhaustive, which poses challenges for robust evaluation with mAP. The
proposed metric, ontology-aware mean average precision (OmAP) addresses the
weaknesses of mAP by utilizing the AudioSet ontology information during the
evaluation. Specifically, we reweight the false positive events in the model
prediction based on the ontology graph distance to the target classes. The OmAP
measure also provides more insights into model performance by evaluations with
different coarse-grained levels in the ontology graph. We conduct human
evaluations and demonstrate that OmAP is more consistent with human perception
than mAP. To further verify the importance of utilizing the ontology
information, we also propose a novel loss function (OBCE) that reweights binary
cross entropy (BCE) loss based on the ontology distance. Our experiment shows
that OBCE can improve both mAP and OmAP metrics on the AudioSet tagging task.

本研究提出了一种新的音频标记任务评估度量 - 本体论感知平均准确率 (OmAP), 该评估度量利用 AudioSet 本体信息，在评估中重新权重模型预测的假阳性事件，提供了更多在本体图中不同粗粒度级别下模型性能的见解，并且比传统的平均准确率 (mAP) 更符合人类感知。此外，本研究还提出了一种新的损失函数 (OBCE), 通过重新加权基于本体距离的二元交叉熵损失 (BCE) 来验证利用本体信息的重要性，实验证明 OBCE 在 AudioSet 标记任务中可以提高 mAP 和 OmAP 的表现。