Large Language Models, such as Generative Pre-trained Transformer 3 (aka.
GPT-3), have been developed to understand language through the analysis of
extensive text data, allowing them to identify patterns and connections between
words. While LLMs have demonstrated impressive performance across various
text-related tasks, they encounter challenges in tasks associated with
reasoning. To address this challenge, Chain of Thought(CoT) prompting method
has been proposed as a means to enhance LLMs' proficiency in complex reasoning
tasks like solving math word problems and answering questions based on logical
argumentative reasoning. The primary aim of this research is to assess how well
four language models can grade reflective essays of third-year medical
students. The assessment will specifically target the evaluation of critical
thinking skills using CoT prompting.
The research will provide the following contributions; to introduce and
educate on the process of instructing models to evaluate reflective essays from
a dataset they have not been previously trained on; to illustrate the use of
CoT prompting as an instructional approach for training large models to carry
out particular tasks. Our results suggest that among all the models, Llama-7b
performs the least effectively, displaying the highest mean squared error.
Conversely, ChatGPT emerges as the superior model, boasting a higher Cohen
kappa score value of 0.53. Lastly, it's important to note that the selected
models do prioritise user privacy by allowing users to delete their own
conducted conversations.

通过 CoT 提示方法来评估四个语言模型对第三年医学生反思性文章的打分，结果显示 Llama-7b 表现最差，均方误差最高，而 ChatGPT 表现出色，科恩卡帕分数达到 0.53。此外，所选模型均优先考虑用户隐私，允许用户删除自己进行的会话。