Introduction: Recently, the effectiveness of Large Language Models (LLMs) has
increased rapidly, allowing them to be used in a great number of applications.
However, the risks posed by the generation of false information through LLMs
significantly limit their applications in sensitive areas such as healthcare,
highlighting the necessity for rigorous validations to determine their utility
and reliability. To date, no study has extensively compared the performance of
LLMs on Polish medical examinations across a broad spectrum of specialties on a
very large dataset. Objectives: This study evaluated the performance of three
Generative Pretrained Transformer (GPT) models on the Polish Board
Certification Exam (Pa\'nstwowy Egzamin Specjalizacyjny, PES) dataset, which
consists of 297 tests. Methods: We developed a software program to download and
process PES exams and tested the performance of GPT models using OpenAI
Application Programming Interface. Results: Our findings reveal that GPT-3.5
did not pass any of the analyzed exams. In contrast, the GPT-4 models
demonstrated the capability to pass the majority of the exams evaluated, with
the most recent model, gpt-4-0125, successfully passing 222 (75%) of them. The
performance of the GPT models varied significantly, displaying excellence in
exams related to certain specialties while completely failing others.
Conclusions: The significant progress and impressive performance of LLM models
hold great promise for the increased application of AI in the field of medicine
in Poland. For instance, this advancement could lead to the development of
AI-based medical assistants for healthcare professionals, enhancing the
efficiency and accuracy of medical services.

通过在 297 个考试中测试 Generative Pretrained Transformer（GPT）模型的性能，研究结果显示 GPT-3.5 没有通过任何考试，而最新的模型 gpt-4-0125 成功通过了 222 个考试（75%）。此研究对于波兰的大型语言模型在医学考试中的性能评估展示了巨大的进展，并展望了将 AI 应用于医疗领域的前景，例如开发基于 AI 的医疗助手来提高医疗服务的效率和准确性。