We present PAPERCLIP (Proposal Abstracts Provide an Effective Representation
for Contrastive Language-Image Pre-training), a method which associates
astronomical observations imaged by telescopes with natural language using a
neural network model. The model is fine-tuned from a pre-trained Contrastive
Language-Image Pre-training (CLIP) model using successful observing proposal
abstracts and corresponding downstream observations, with the abstracts
optionally summarized via guided generation using large language models (LLMs).
Using observations from the Hubble Space Telescope (HST) as an example, we show
that the fine-tuned model embodies a meaningful joint representation between
observations and natural language through tests targeting image retrieval
(i.e., finding the most relevant observations using natural language queries)
and description retrieval (i.e., querying for astrophysical object classes and
use cases most relevant to a given observation). Our study demonstrates the
potential for using generalist foundation models rather than task-specific
models for interacting with astronomical data by leveraging text as an
interface.

我们提出了一种名为 PAPERCLIP 的方法，使用神经网络模型将望远镜拍摄的天文观测与自然语言进行关联，通过从预训练的对比语言 - 图像预训练（CLIP）模型微调成功的观测提案摘要和相应的下游观测，可选择使用大型语言模型（LLM）来进行摘要总结。以哈勃空间望远镜（HST）的观测为例，我们展示了经过微调的模型通过针对图像检索和描述检索的测试体现了观测和自然语言之间的有意义的联合表示，通过使用文本作为界面，我们的研究证明了利用通用基础模型而非任务特定模型与天文数据交互的潜力。