Due to deteriorating environmental conditions and increasing human activity,
conservation efforts directed towards wildlife is crucial. Motion-activated
camera traps constitute an efficient tool for tracking and monitoring wildlife
populations across the globe. Supervised learning techniques have been
successfully deployed to analyze such imagery, however training such techniques
requires annotations from experts. Reducing the reliance on costly labelled
data therefore has immense potential in developing large-scale wildlife
tracking solutions with markedly less human labor. In this work we propose
WildMatch, a novel zero-shot species classification framework that leverages
multimodal foundation models. In particular, we instruction tune
vision-language models to generate detailed visual descriptions of camera trap
images using similar terminology to experts. Then, we match the generated
caption to an external knowledge base of descriptions in order to determine the
species in a zero-shot manner. We investigate techniques to build instruction
tuning datasets for detailed animal description generation and propose a novel
knowledge augmentation technique to enhance caption quality. We demonstrate the
performance of WildMatch on a new camera trap dataset collected in the
Magdalena Medio region of Colombia.

通过减少对昂贵的标记数据的依赖，我们提出了一种新的零样本物种分类方法 WildMatch，该方法使用多模态基础模型，利用人们熟悉的术语生成摄像机陷阱图像的详细视觉描述，并将生成的描述与外部知识库中的描述进行匹配，以零样本的方式确定物种。我们还研究了构建详细动物描述生成的指导调优数据集的技术，并提出了一种新的知识增强技术来提高描述质量。我们在哥伦比亚 Magdalena Medio 地区的一个新的摄像机陷阱数据集上展示了 WildMatch 的性能。