We present AstroCLIP, a strategy to facilitate the construction of
astronomical foundation models that bridge the gap between diverse
observational modalities. We demonstrate that a cross-modal contrastive
learning approach between images and optical spectra of galaxies yields highly
informative embeddings of both modalities. In particular, we apply our method
on multi-band images and optical spectra from the Dark Energy Spectroscopic
Instrument (DESI), and show that: (1) these embeddings are well-aligned between
modalities and can be used for accurate cross-modal searches, and (2) these
embeddings encode valuable physical information about the galaxies -- in
particular redshift and stellar mass -- that can be used to achieve competitive
zero- and few- shot predictions without further finetuning. Additionally, in
the process of developing our approach, we also construct a novel,
transformer-based model and pretraining approach for processing galaxy spectra.

AstroCLIP 是一种构建天文基础模型的策略，它通过图像和光学星系光谱之间的跨模态对比学习方法实现了高度信息密集的嵌入表达，从而实现了准确的跨模态搜索和有竞争力的零式和少样本预测。