Modern plankton high-throughput monitoring relies on deep learning
classifiers for species recognition in water ecosystems. Despite satisfactory
nominal performances, a significant challenge arises from Dataset Shift, which
causes performances to drop during deployment. In our study, we integrate the
ZooLake dataset with manually-annotated images from 10 independent days of
deployment, serving as test cells to benchmark Out-Of-Dataset (OOD)
performances. Our analysis reveals instances where classifiers, initially
performing well in In-Dataset conditions, encounter notable failures in
practical scenarios. For example, a MobileNet with a 92% nominal test accuracy
shows a 77% OOD accuracy. We systematically investigate conditions leading to
OOD performance drops and propose a preemptive assessment method to identify
potential pitfalls when classifying new data, and pinpoint features in OOD
images that adversely impact classification. We present a three-step pipeline:
(i) identifying OOD degradation compared to nominal test performance, (ii)
conducting a diagnostic analysis of degradation causes, and (iii) providing
solutions. We find that ensembles of BEiT vision transformers, with targeted
augmentations addressing OOD robustness, geometric ensembling, and
rotation-based test-time augmentation, constitute the most robust model, which
we call BEsT model. It achieves an 83% OOD accuracy, with errors concentrated
on container classes. Moreover, it exhibits lower sensitivity to dataset shift,
and reproduces well the plankton abundances. Our proposed pipeline is
applicable to generic plankton classifiers, contingent on the availability of
suitable test cells. By identifying critical shortcomings and offering
practical procedures to fortify models against dataset shift, our study
contributes to the development of more reliable plankton classification
technologies.

现代浮游生物高通量监测依赖于深度学习分类器对水生态系统中的物种识别。我们研究中集成了 ZooLake 数据集和 10 个独立部署日的手动标注图像，作为用于评估超出数据集的表现的测试单元。我们通过分析发现，在实际场景中，分类器在初始数据集条件下表现良好的情况下也会遇到显著失败的例子。我们提出了一个三步骤的流程：（一）辨别比正常测试性能下的超出数据集降级，（二）进行降级原因的诊断分析，以及（三）提供解决方案。我们发现，使用 BEiT 视觉转换器进行集成的模型在超出数据集鲁棒性、几何集合和基于旋转的测试时间增强方面具有较强的模型，我们称之为 BEsT 模型。它的超出数据集准确度为 83％，错误集中在容器类上。此外，它对数据集变化的敏感性较低，并且能够很好地体现浮游生物的丰度。我们提出的流程适用于通用的浮游生物分类器，在适当的测试单元可用的情况下。通过非常短的表达，我们的研究为更可靠的浮游生物分类技术的发展提供了帮助。