The recent integration of visual capabilities into Large Language Models
(LLMs) has the potential to play a pivotal role in science and technology
education, where visual elements such as diagrams, charts, and tables are
commonly used to improve the learning experience. This study investigates the
performance of ChatGPT-4 Vision, OpenAI's most advanced visual model at the
time the study was conducted, on the Bachelor in Computer Science section of
Brazil's 2021 National Undergraduate Exam (ENADE). By presenting the model with
the exam's open and multiple-choice questions in their original image format
and allowing for reassessment in response to differing answer keys, we were
able to evaluate the model's reasoning and self-reflecting capabilities in a
large-scale academic assessment involving textual and visual content. ChatGPT-4
Vision significantly outperformed the average exam participant, positioning
itself within the top 10 best score percentile. While it excelled in questions
that incorporated visual elements, it also encountered challenges with question
interpretation, logical reasoning, and visual acuity. The involvement of an
independent expert panel to review cases of disagreement between the model and
the answer key revealed some poorly constructed questions containing vague or
ambiguous statements, calling attention to the critical need for improved
question design in future exams. Our findings suggest that while ChatGPT-4
Vision shows promise in multimodal academic evaluations, human oversight
remains crucial for verifying the model's accuracy and ensuring the fairness of
high-stakes educational exams. The paper's research materials are publicly
available at this https URL

ChatGPT-4 Vision 在巴西 2021 年本科国家考试中展现了优秀的视觉能力，但在问题解释、逻辑推理和视觉敏锐度方面遇到了困难，提示未来考试需要改进问题设计。研究结果表明，虽然 ChatGPT-4 Vision 在多模态学术评估中表现出潜力，但人类监督仍然至关重要，以验证模型的准确性并确保高风险教育考试的公平性。

评估 ChatGPT-4 Vision 在巴西国家本科计算机科学考试中

Evaluating ChatGPT-4 Vision on Brazil's National Undergraduate Computer  Science Exam

This study explores the robustness of university assessments against the use
of Open AI's Generative Pre-Trained Transformer 4 (GPT-4) generated content and
evaluates the ability of academic staff to detect its use when supported by the
Turnitin Artificial Intelligence (AI) detection tool. The research involved
twenty-two GPT-4 generated submissions being created and included in the
assessment process to be marked by fifteen different faculty members. The study
reveals that although the detection tool identified 91% of the experimental
submissions as containing some AI-generated content, the total detected content
was only 54.8%. This suggests that the use of adversarial techniques regarding
prompt engineering is an effective method in evading AI detection tools and
highlights that improvements to AI detection software are needed. Using the
Turnitin AI detect tool, faculty reported 54.5% of the experimental submissions
to the academic misconduct process, suggesting the need for increased awareness
and training into these tools. Genuine submissions received a mean score of
54.4, whereas AI-generated content scored 52.3, indicating the comparable
performance of GPT-4 in real-life situations. Recommendations include adjusting
assessment strategies to make them more resistant to the use of AI tools, using
AI-inclusive assessment where possible, and providing comprehensive training
programs for faculty and students. This research contributes to understanding
the relationship between AI-generated content and academic assessment, urging
further investigation to preserve academic integrity.

本研究使用 GPT-4 生成内容来测试其对大学评估的鲁棒性，并评估学术人员在 Turnitin AI 检测工具支持下是否能够检测到其使用。研究表明，使用对抗技术可以有效地逃避 AI 检测工具的检测，并提出了应调整评估策略、进行全面培训等建议。