Large language models (LLMs) have transformed AI and achieved breakthrough
performance on a wide range of tasks that require human intelligence. In
science, perhaps the most interesting application of LLMs is for hypothesis
formation. A feature of LLMs, which results from their probabilistic structure,
is that the output text is not necessarily a valid inference from the training
text. These are 'hallucinations', and are a serious problem in many
applications. However, in science, hallucinations may be useful: they are novel
hypotheses whose validity may be tested by laboratory experiments. Here we
experimentally test the use of LLMs as a source of scientific hypotheses using
the domain of breast cancer treatment. We applied the LLM GPT4 to hypothesize
novel pairs of FDA-approved non-cancer drugs that target the MCF7 breast cancer
cell line relative to the non-tumorigenic breast cell line MCF10A. In the first
round of laboratory experiments GPT4 succeeded in discovering three drug
combinations (out of 12 tested) with synergy scores above the positive
controls. These combinations were itraconazole + atenolol, disulfiram +
simvastatin and dipyridamole + mebendazole. GPT4 was then asked to generate new
combinations after considering its initial results. It then discovered three
more combinations with positive synergy scores (out of four tested), these were
disulfiram + fulvestrant, mebendazole + quinacrine and disulfiram + quinacrine.
A limitation of GPT4 as a generator of hypotheses was that its explanations for
them were formulaic and unconvincing. We conclude that LLMs are an exciting
novel source of scientific hypotheses.

通过实验测试，我们发现大型语言模型是一种令人兴奋的新型科学假设来源，并成功应用于乳腺癌治疗领域来生成新的、与非肿瘤性乳腺细胞系 MCF10A 相比针对 MCF7 乳腺癌细胞系的 FDA 批准的非癌症药物组合，并发现多个具有积极协同效应的药物组合。

大型语言模型进行科学假设生成的实验室验证：在乳腺癌治疗中的应用

Scientific Hypothesis Generation by a Large Language Model: Laboratory  Validation in Breast Cancer Treatment

Machine learning (ML) models that achieve high average accuracy can still
underperform on semantically coherent subsets (i.e. "slices") of data. This
behavior can have significant societal consequences for the safety or bias of
the model in deployment, but identifying these underperforming slices can be
difficult in practice, especially in domains where practitioners lack access to
group annotations to define coherent subsets of their data. Motivated by these
challenges, ML researchers have developed new slice discovery algorithms that
aim to group together coherent and high-error subsets of data. However, there
has been little evaluation focused on whether these tools help humans form
correct hypotheses about where (for which groups) their model underperforms. We
conduct a controlled user study (N = 15) where we show 40 slices output by two
state-of-the-art slice discovery algorithms to users, and ask them to form
hypotheses about where an object detection model underperforms. Our results
provide positive evidence that these tools provide some benefit over a naive
baseline, and also shed light on challenges faced by users during the
hypothesis formation step. We conclude by discussing design opportunities for
ML and HCI researchers. Our findings point to the importance of centering users
when designing and evaluating new tools for slice discovery.

机器学习中的 slice discovery algorithm 对于模型表现下降等问题提供了一种新方法，相关的使用者研究表明这种方法比 naïve baseline 更好，但是在假设形成的环节上仍然存在挑战，因此在设计和评估新的工具时需要关注使用者。