IMPORTANCE The response effectiveness of different large language models
(LLMs) and various individuals, including medical students, graduate students,
and practicing physicians, in pediatric ophthalmology consultations, has not
been clearly established yet. OBJECTIVE Design a 100-question exam based on
pediatric ophthalmology to evaluate the performance of LLMs in highly
specialized scenarios and compare them with the performance of medical students
and physicians at different levels. DESIGN, SETTING, AND PARTICIPANTS This
survey study assessed three LLMs, namely ChatGPT (GPT-3.5), GPT-4, and PaLM2,
were assessed alongside three human cohorts: medical students, postgraduate
students, and attending physicians, in their ability to answer questions
related to pediatric ophthalmology. It was conducted by administering
questionnaires in the form of test papers through the LLM network interface,
with the valuable participation of volunteers. MAIN OUTCOMES AND MEASURES Mean
scores of LLM and humans on 100 multiple-choice questions, as well as the
answer stability, correlation, and response confidence of each LLM. RESULTS
GPT-4 performed comparably to attending physicians, while ChatGPT (GPT-3.5) and
PaLM2 outperformed medical students but slightly trailed behind postgraduate
students. Furthermore, GPT-4 exhibited greater stability and confidence when
responding to inquiries compared to ChatGPT (GPT-3.5) and PaLM2. CONCLUSIONS
AND RELEVANCE Our results underscore the potential for LLMs to provide medical
assistance in pediatric ophthalmology and suggest significant capacity to guide
the education of medical students.

评估大型语言模型在儿科眼科学中的表现与医学生和医生的对比试验结果显示，大型语言模型有潜力为儿科眼科提供医疗援助，并具有指导医学生教育的重要能力。