We present a new benchmark dataset, Sapsucker Woods 60 (SSW60), for advancing
research on audiovisual fine-grained categorization. While our community has
made great strides in fine-grained visual categorization on images, the
counterparts in audio and video fine-grained categorization are relatively
unexplored. To encourage advancements in this space, we have carefully
constructed the SSW60 dataset to enable researchers to experiment with
classifying the same set of categories in three different modalities: images,
audio, and video. The dataset covers 60 species of birds and is comprised of
images from existing datasets, and brand new, expert-curated audio and video
datasets. We thoroughly benchmark audiovisual classification performance and
modality fusion experiments through the use of state-of-the-art transformer
methods. Our findings show that performance of audiovisual fusion methods is
better than using exclusively image or audio based methods for the task of
video classification. We also present interesting modality transfer
experiments, enabled by the unique construction of SSW60 to encompass three
different modalities. We hope the SSW60 dataset and accompanying baselines spur
research in this fascinating area.

本文介绍了一种新的基准数据集 SSW60，旨在促进音视频细粒度分类研究，覆盖 60 种鸟类，可用于图像、音频和视频分类，实验结果表明，使用音视频融合方法比单独使用图像或音频方法的性能更好，同时进行了有趣的模态转换实验。