In this paper, we explore the challenges inherent to Large Language Models (LLMs) like GPT-4, particularly their propensity for hallucinations, logic mistakes, and incorrect conclusions when tasked with answering complex questions. The capacity of LLMs to present erroneous answers in a coherent and semantically rigorous manner further complicates the detection of factual inaccuracies. This issue is especially pronounced in fields that require specialized expertise. Our work delves into these challenges, aiming to enhance the understanding and mitigation of such errors, thereby contributing to the improvement of LLM accuracy and reliability in scientific and other specialized domains. Our findings reveal a non-linear relationship between the context's relevancy and the answers' measured quality. In addition, we demonstrate that with the correct calibration, it is possible to automate the grading procedure -- a finding suggesting that, at least to some degree, the LLMs can be used to self-examine the quality of their own performance. Finally, we describe an experimental platform that can be seen as a proof-of-concept of the techniques described in this work.

本文研究了大型语言模型（如GPT-4）存在的挑战，特别是在回答复杂问题时容易出现错觉、逻辑错误和错误结论的问题。语言模型能以一种连贯且语义严谨的方式呈现错误答案的能力进一步增加了发现事实不准确性的困难。这个问题在需要专业知识的领域尤为明显。我们的工作深入探讨了这些挑战，旨在增强对此类错误的理解和缓解，从而提高科学和其他专业领域中LLMs的准确性和可靠性。我们的研究结果揭示了上下文相关性与答案质量之间的非线性关系。此外，我们证明了在正确校准的情况下，可能实现自动评分——这一发现表明，至少在某种程度上，LLMs可以用于自我检验其性能质量。最后，我们描述了一个实验平台，可以被视为对本文描述的技术的概念验证。

背景问题：为科学应用数据高效扩充的大型语言模型