Large language models (LLMs) have been extensively studied for their
abilities to generate convincing natural language sequences, however their
utility for quantitative information retrieval is less well understood. In this
paper we explore the feasibility of LLMs as a mechanism for quantitative
knowledge retrieval to aid data analysis tasks such as elicitation of prior
distributions for Bayesian models and imputation of missing data. We present a
prompt engineering framework, treating an LLM as an interface to a latent space
of scientific literature, comparing responses in different contexts and domains
against more established approaches. Implications and challenges of using LLMs
as 'experts' are discussed.

大型语言模型可用于定量信息检索，以帮助数据分析任务，如贝叶斯模型的先验分布以及缺失数据的插补。我们提出了一个提示工程框架，将大型语言模型视为科学文献的潜在空间界面，并与其他已建立的方法进行比较。同时讨论了使用大型语言模型作为 “专家” 的影响和挑战。

大型语言模型的数量化知识检索

Quantitative knowledge retrieval from large language models

This paper presents a comprehensive evaluation of the code generation
capabilities of ChatGPT, a prominent large language model, compared to human
programmers. A novel dataset of 131 code-generation prompts across 5 categories
was curated to enable robust analysis. Code solutions were generated by both
ChatGPT and humans for all prompts, resulting in 262 code samples. A meticulous
manual assessment methodology prioritized evaluating correctness,
comprehensibility, and security using 14 established code quality metrics. The
key findings reveal ChatGPT's strengths in crafting concise, efficient code
with advanced constructs, showcasing strengths in data analysis tasks (93.1%
accuracy) but limitations in visual-graphical challenges. Comparative analysis
with human code highlights ChatGPT's inclination towards modular design and
superior error handling. Additionally, machine learning models effectively
distinguished ChatGPT from human code with up to 88% accuracy, suggesting
detectable coding style disparities. By providing profound insights into
ChatGPT's code generation capabilities and limitations through quantitative
metrics and qualitative analysis, this study makes valuable contributions
toward advancing AI-based programming assistants. The curated dataset and
methodology offer a robust foundation for future research in this nascent
domain. All data and codes are available on
this https URL

ChatGPT 是一种显著的大型语言模型，通过与人类程序员进行比较，本文提出了对其代码生成能力的综合评估。通过构建一个包含 5 个类别的新颖数据集，共计 131 个代码生成提示，ChatGPT 和人类程序员生成了 262 个代码样例。使用 14 个已建立的代码质量度量方法进行详细的手动评估，重点评估了正确性、可理解性和安全性。关键发现揭示了 ChatGPT 在制作简洁高效、具有高级结构的代码方面的优势，并展示了在数据分析任务（93.1% 准确度）中的强项，但在视觉图形方面存在局限性。与人类代码的比较分析凸显了 ChatGPT 对模块化设计和优秀的错误处理的倾向。此外，机器学习模型可以有效地区分 ChatGPT 和人类代码，准确率高达 88%，表明可以检测到的编码风格差异。通过量化指标和定性分析深入探讨了 ChatGPT 的代码生成能力和局限性，为推进基于人工智能的编程助手提供了宝贵的见解。精心策划的数据集和方法为这个新兴领域的未来研究提供了坚实的基础。所有数据和代码都可在此 https URL 上找到。