In-context learning$\unicode{x2013}$the ability to configure a model's behavior with different prompts$\unicode{x2013}$has revolutionized the field of natural language processing, alleviating the need for task-specific models and paving the way for generalist models capable of assisting with any query. Computer vision, in contrast, has largely stayed in the former regime: specialized decoders and finetuning protocols are generally required to perform dense tasks such as semantic segmentation and depth estimation. In this work we explore a simple mechanism for in-context learning of such scene understanding tasks: nearest neighbor retrieval from a prompt of annotated features. We propose a new pretraining protocol$\unicode{x2013}$leveraging attention within and across images$\unicode{x2013}$which yields representations particularly useful in this regime. The resulting Hummingbird model, suitably prompted, performs various scene understanding tasks without modification while approaching the performance of specialists that have been finetuned for each task. Moreover, Hummingbird can be configured to perform new tasks much more efficiently than finetuned models, raising the possibility of scene understanding in the interactive assistant regime.

探索了一种简单的机制，即通过从带注释的特征提示中进行最近邻检索的上下文学习，并提出了一种新的预训练协议，利用图像内部和跨图像的注意力，使得Hummingbird模型在不修改的情况下能够执行各种场景理解任务，同时接近于为每个任务进行微调的专家的性能，并且可以更高效地配置Hummingbird模型以执行新任务，进一步提高了交互式助手模式下的场景理解的可能性。

基于情境的场景理解