Recent years witnessed an increase in the amount of research on the task of
Question Difficulty Estimation from Text QDET with Natural Language Processing
(NLP) techniques, with the goal of targeting the limitations of traditional
approaches to question calibration. However, almost the entirety of previous
research focused on single silos, without performing quantitative comparisons
between different models or across datasets from different educational domains.
In this work, we aim at filling this gap, by quantitatively analyzing several
approaches proposed in previous research, and comparing their performance on
three publicly available real world datasets containing questions of different
types from different educational domains. Specifically, we consider reading
comprehension Multiple Choice Questions (MCQs), science MCQs, and math
questions. We find that Transformer based models are the best performing across
different educational domains, with DistilBERT performing almost as well as
BERT, and that they outperform other approaches even on smaller datasets. As
for the other models, the hybrid ones often outperform the ones based on a
single type of features, the ones based on linguistic features perform well on
reading comprehension questions, while frequency based features (TF-IDF) and
word embeddings (word2vec) perform better in domain knowledge assessment.

本研究通过定量分析之前研究中提出的几种方法，比较它们在三个不同教育领域包含不同类型问题的公开真实数据集上的表现，发现基于 Transformer 的模型在跨不同教育领域的 MCQs、科学题和数学题等方面表现最佳，超越其他方法甚至在较小的数据集上也表现得更好。