Biomedical image analysis is fundamental for biomedical discovery in cell
biology, pathology, radiology, and many other biomedical domains. Holistic
image analysis comprises interdependent subtasks such as segmentation,
detection, and recognition of relevant objects. Here, we propose BiomedParse, a
biomedical foundation model for imaging parsing that can jointly conduct
segmentation, detection, and recognition for 82 object types across 9 imaging
modalities. Through joint learning, we can improve accuracy for individual
tasks and enable novel applications such as segmenting all relevant objects in
an image through a text prompt, rather than requiring users to laboriously
specify the bounding box for each object. We leveraged readily available
natural-language labels or descriptions accompanying those datasets and use
GPT-4 to harmonize the noisy, unstructured text information with established
biomedical object ontologies. We created a large dataset comprising over six
million triples of image, segmentation mask, and textual description. On image
segmentation, we showed that BiomedParse is broadly applicable, outperforming
state-of-the-art methods on 102,855 test image-mask-label triples across 9
imaging modalities (everything). On object detection, which aims to locate a
specific object of interest, BiomedParse again attained state-of-the-art
performance, especially on objects with irregular shapes (everywhere). On
object recognition, which aims to identify all objects in a given image along
with their semantic types, we showed that BiomedParse can simultaneously
segment and label all biomedical objects in an image (all at once). In summary,
BiomedParse is an all-in-one tool for biomedical image analysis by jointly
solving segmentation, detection, and recognition for all major biomedical image
modalities, paving the path for efficient and accurate image-based biomedical
discovery.

BiomedParse 是一个用于生物医学图像分析的全能工具，通过联合解决分割、检测和识别等主要生物医学图像模态，为高效准确的基于图像的生物医学发现铺平了道路。