This study investigates GPT-4's assessment of its performance in healthcare
applications. A simple prompting technique was used to prompt the LLM with
questions taken from the United States Medical Licensing Examination (USMLE)
questionnaire and it was tasked to evaluate its confidence score before posing
the question and after asking the question. The questionnaire was categorized
into two groups-questions with feedback (WF) and questions with no feedback(NF)
post-question. The model was asked to provide absolute and relative confidence
scores before and after each question. The experimental findings were analyzed
using statistical tools to study the variability of confidence in WF and NF
groups. Additionally, a sequential analysis was conducted to observe the
performance variation for the WF and NF groups. Results indicate that feedback
influences relative confidence but doesn't consistently increase or decrease
it. Understanding the performance of LLM is paramount in exploring its utility
in sensitive areas like healthcare. This study contributes to the ongoing
discourse on the reliability of AI, particularly of LLMs like GPT-4, within
healthcare, offering insights into how feedback mechanisms might be optimized
to enhance AI-assisted medical education and decision support.

本研究探讨了 GPT-4 在医疗应用中的性能评估，使用简单的提示技术从美国医学执照考试问卷中提取问题作为提示，任务是在提问前和提问后评估它的置信度得分。分为有反馈和无反馈两组问题进行分类，实验结果表明反馈会影响相对置信度但并不一致地增加或减少。这项研究对于 AI 可靠性的讨论具有重要意义，特别是在医疗领域中的 LLM 类模型（如 GPT-4），为优化反馈机制以提升 AI 辅助医学教育和决策支持提供了有价值的见解。