Understanding the way cells communicate, co-locate, and interrelate is
essential to understanding human physiology. Hematoxylin and eosin (H&E)
staining is ubiquitously available both for clinical studies and research. The
Colon Nucleus Identification and Classification (CoNIC) Challenge has recently
innovated on robust artificial intelligence labeling of six cell types on H&E
stains of the colon. However, this is a very small fraction of the number of
potential cell classification types. Specifically, the CoNIC Challenge is
unable to classify epithelial subtypes (progenitor, endocrine, goblet),
lymphocyte subtypes (B, helper T, cytotoxic T), or connective subtypes
(fibroblasts, stromal). In this paper, we propose to use inter-modality
learning to label previously un-labelable cell types on virtual H&E. We
leveraged multiplexed immunofluorescence (MxIF) histology imaging to identify
14 subclasses of cell types. We performed style transfer to synthesize virtual
H&E from MxIF and transferred the higher density labels from MxIF to these
virtual H&E images. We then evaluated the efficacy of learning in this
approach. We identified helper T and progenitor nuclei with positive predictive
values of $0.34 \pm 0.15$ (prevalence $0.03 \pm 0.01$) and $0.47 \pm 0.1$
(prevalence $0.07 \pm 0.02$) respectively on virtual H&E. This approach
represents a promising step towards automating annotation in digital pathology.

利用多模态学习在数字病理学中为先前无法分类的细胞类型自动标注，代表着自动标注数字病理学的有希望的进展。

核子亚型分类的跨模态学习

Nucleus subtype classification using inter-modality learning

By supporting multi-modal retrieval training and evaluation, image captioning
datasets have spurred remarkable progress on representation learning.
Unfortunately, datasets have limited cross-modal associations: images are not
paired with other images, captions are only paired with other captions of the
same image, there are no negative associations and there are missing positive
cross-modal associations. This undermines research into how inter-modality
learning impacts intra-modality tasks. We address this gap with Crisscrossed
Captions (CxC), an extension of the MS-COCO dataset with human semantic
similarity judgments for 267,095 intra- and inter-modality pairs. We report
baseline results on CxC for strong existing unimodal and multimodal models. We
also evaluate a multitask dual encoder trained on both image-caption and
caption-caption pairs that crucially demonstrates CxC's value for measuring the
influence of intra- and inter-modality learning.

本研究扩展了 MS-COCO 数据集，提出了 Crisscrossed Captions (CxC)，用于评估跨模态训练和评估，可用于研究跨模态学习对内模态任务的影响，并对现有内模态和多模态模型进行了基准测试。