Evaluating multiple-choice questions (MCQs) involves either labor intensive
human assessments or automated methods that prioritize readability, often
overlooking deeper question design flaws. To address this issue, we introduce
the Scalable Automatic Question Usability Evaluation Toolkit (SAQUET), an
open-source tool that leverages the Item-Writing Flaws (IWF) rubric for a
comprehensive and automated quality evaluation of MCQs. By harnessing the
latest in large language models such as GPT-4, advanced word embeddings, and
Transformers designed to analyze textual complexity, SAQUET effectively
pinpoints and assesses a wide array of flaws in MCQs. We first demonstrate the
discrepancy between commonly used automated evaluation metrics and the human
assessment of MCQ quality. Then we evaluate SAQUET on a diverse dataset of MCQs
across the five domains of Chemistry, Statistics, Computer Science, Humanities,
and Healthcare, showing how it effectively distinguishes between flawed and
flawless questions, providing a level of analysis beyond what is achievable
with traditional metrics. With an accuracy rate of over 94% in detecting the
presence of flaws identified by human evaluators, our findings emphasize the
limitations of existing evaluation methods and showcase potential in improving
the quality of educational assessments.

Evaluating multiple-choice questions often overlooks deeper question design flaws, but the SAQUET tool effectively automates the assessment process using large language models, word embeddings, and Transformers, achieving a high accuracy rate in detecting flaws and showcasing potential for improving educational assessments.

自动问卷可用性评估工具

An Automatic Question Usability Evaluation Toolkit

The rapid adoption of Generative Artificial Intelligence (GenAI) technologies
in higher education has raised concerns about academic integrity, assessment
practices, and student learning. Banning or blocking GenAI tools has proven
ineffective, and punitive approaches ignore the potential benefits of these
technologies. This paper presents the findings of a pilot study conducted at
British University Vietnam (BUV) exploring the implementation of the Artificial
Intelligence Assessment Scale (AIAS), a flexible framework for incorporating
GenAI into educational assessments. The AIAS consists of five levels, ranging
from 'No AI' to 'Full AI', enabling educators to design assessments that focus
on areas requiring human input and critical thinking.
Following the implementation of the AIAS, the pilot study results indicate a
significant reduction in academic misconduct cases related to GenAI, a 5.9%
increase in student attainment across the university, and a 33.3% increase in
module passing rates. The AIAS facilitated a shift in pedagogical practices,
with faculty members incorporating GenAI tools into their modules and students
producing innovative multimodal submissions. The findings suggest that the AIAS
can support the effective integration of GenAI in HE, promoting academic
integrity while leveraging the technology's potential to enhance learning
experiences.

英国越南大学举行的一项试点研究结果表明，将生成人工智能（GenAI）技术纳入教育评估中，在提高学习体验的同时促进学术诚信，并降低与 GenAI 相关的学术不端行为，使学生的成绩和模块通过率都得到了显著提升。