The recent integration of visual capabilities into Large Language Models
(LLMs) has the potential to play a pivotal role in science and technology
education, where visual elements such as diagrams, charts, and tables are
commonly used to improve the learning experience. This study investigates the
performance of ChatGPT-4 Vision, OpenAI's most advanced visual model at the
time the study was conducted, on the Bachelor in Computer Science section of
Brazil's 2021 National Undergraduate Exam (ENADE). By presenting the model with
the exam's open and multiple-choice questions in their original image format
and allowing for reassessment in response to differing answer keys, we were
able to evaluate the model's reasoning and self-reflecting capabilities in a
large-scale academic assessment involving textual and visual content. ChatGPT-4
Vision significantly outperformed the average exam participant, positioning
itself within the top 10 best score percentile. While it excelled in questions
that incorporated visual elements, it also encountered challenges with question
interpretation, logical reasoning, and visual acuity. The involvement of an
independent expert panel to review cases of disagreement between the model and
the answer key revealed some poorly constructed questions containing vague or
ambiguous statements, calling attention to the critical need for improved
question design in future exams. Our findings suggest that while ChatGPT-4
Vision shows promise in multimodal academic evaluations, human oversight
remains crucial for verifying the model's accuracy and ensuring the fairness of
high-stakes educational exams. The paper's research materials are publicly
available at this https URL

ChatGPT-4 Vision 在巴西 2021 年本科国家考试中展现了优秀的视觉能力，但在问题解释、逻辑推理和视觉敏锐度方面遇到了困难，提示未来考试需要改进问题设计。研究结果表明，虽然 ChatGPT-4 Vision 在多模态学术评估中表现出潜力，但人类监督仍然至关重要，以验证模型的准确性并确保高风险教育考试的公平性。