In order for artificial agents to successfully perform tasks in changing
environments, they must be able to both detect and adapt to novelty. However,
visual novelty detection research often only evaluates on repurposed datasets
such as CIFAR-10 originally intended for object classification, where images
focus on one distinct, well-centered object. New benchmarks are needed to
represent the challenges of navigating the complex scenes of an open world. Our
new NovelCraft dataset contains multimodal episodic data of the images and
symbolic world-states seen by an agent completing a pogo stick assembly task
within a modified Minecraft environment. In some episodes, we insert novel
objects within the complex 3D scene that may impact gameplay and appear in a
variety of sizes and positions. Our visual novelty detection benchmark finds
that methods that rank best on popular area-under-the-curve metrics may be
outperformed by simpler alternatives when controlling false positives matters
most. Further multi-modal novelty detection experiments suggest that methods
that fuse both visual and symbolic information can improve time until detection
as well as overall discrimination. Finally, our evaluation of recent
generalized category discovery methods suggests that adapting to new imbalanced
categories in complex scenes remains an exciting open problem.

本研究介绍了一个新的基准数据集 NovelCraft，包含在 Minecraft 环境中完成跳跃球组装任务的代理人所看到的图像和符号世界状态的多模态故事数据。进一步的多模态新颖性检测实验表明，融合视觉和象征信息的方法能够提高检测时间和总体识别能力，同时对于复杂场景中新不平衡分类的适应仍然是一个令人兴奋的开放性问题。