Large-scale visual-language pre-trained models (VLPM) have proven their
excellent performance in downstream object detection for natural scenes.
However, zero-shot nuclei detection on H\&E images via VLPMs remains
underexplored. The large gap between medical images and the web-originated
text-image pairs used for pre-training makes it a challenging task. In this
paper, we attempt to explore the potential of the object-level VLPM, Grounded
Language-Image Pre-training (GLIP) model, for zero-shot nuclei detection.
Concretely, an automatic prompts design pipeline is devised based on the
association binding trait of VLPM and the image-to-text VLPM BLIP, avoiding
empirical manual prompts engineering. We further establish a self-training
framework, using the automatically designed prompts to generate the preliminary
results as pseudo labels from GLIP and refine the predicted boxes in an
iterative manner. Our method achieves a remarkable performance for label-free
nuclei detection, surpassing other comparison methods. Foremost, our work
demonstrates that the VLPM pre-trained on natural image-text pairs exhibits
astonishing potential for downstream tasks in the medical field as well. Code
will be released at this https URL

该论文探讨了如何使用以大规模自然图像文本对为预训练基础的 VLPM 模型，在医学图像检测中实现零样本细胞核检测，并提出了一种基于自动提示设计管道的框架。通过自我训练，该方法在无标签的情况下表现出优异的细胞核检测性能，并展示了 VLPM 在医学领域中的巨大潜力。