Distribution shift is a common situation in machine learning tasks, where the
data used for training a model is different from the data the model is applied
to in the real world. This issue arises across multiple technical settings:
from standard prediction tasks, to time-series forecasting, and to more recent
applications of large language models (LLMs). This mismatch can lead to
performance reductions, and can be related to a multiplicity of factors:
sampling issues and non-representative data, changes in the environment or
policies, or the emergence of previously unseen scenarios. This brief focuses
on the definition and detection of distribution shifts in educational settings.
We focus on standard prediction problems, where the task is to learn a model
that takes in a series of input (predictors) $X=(x_1,x_2,...,x_m)$ and produces
an output $Y=f(X)$.

在机器学习任务中，分布偏移是一种常见情况，其表示训练模型使用的数据与实际应用模型的数据不同。本文旨在定义和检测教育环境中的分布偏移，关注标准预测问题，即学习一个以输入序列为输入（预测变量）X=(x_1,x_2,...,x_m) 并生成输出 Y=f (X) 的模型。

公正中心技术简报：分布偏移的定义和检测

Fairness Hub Technical Briefs: Definition and Detection of Distribution  Shift

Evaluating open-ended written examination responses from students is an
essential yet time-intensive task for educators, requiring a high degree of
effort, consistency, and precision. Recent developments in Large Language
Models (LLMs) present a promising opportunity to balance the need for thorough
evaluation with efficient use of educators' time. In our study, we explore the
effectiveness of LLMs ChatGPT-3.5, ChatGPT-4, Claude-3, and Mistral-Large in
assessing university students' open-ended answers to questions made about
reference material they have studied. Each model was instructed to evaluate 54
answers repeatedly under two conditions: 10 times (10-shot) with a temperature
setting of 0.0 and 10 times with a temperature of 0.5, expecting a total of
1,080 evaluations per model and 4,320 evaluations across all models. The RAG
(Retrieval Augmented Generation) framework was used as the framework to make
the LLMs to process the evaluation of the answers. As of spring 2024, our
analysis revealed notable variations in consistency and the grading outcomes
provided by studied LLMs. There is a need to comprehend strengths and
weaknesses of LLMs in educational settings for evaluating open-ended written
responses. Further comparative research is essential to determine the accuracy
and cost-effectiveness of using LLMs for educational assessments.

教育工作者评估开放式书面考试答案是一项需要大量精力、一致性和准确性的重要任务。本研究探索了大型语言模型在评估大学生对参考资料提出的开放式问题的答案时的效果，发现 LLMs 的一致性和评分结果存在显著差异。进一步的比较研究对于确定使用 LLMs 进行教育评估的准确性和成本效益至关重要。

使用 LLMs 评估学生的开放式书面答案：基于 RAG 框架，针对 GPT-3.5，GPT-4，Claude-3 和 Mistral-Large 进行

Evaluating Students' Open-ended Written Responses with LLMs: Using the  RAG Framework for GPT-3.5, GPT-4, Claude-3, and Mistral-Large

Emerging technologies like generative AI tools, including ChatGPT, are
increasingly utilized in educational settings, offering innovative approaches
to learning while simultaneously posing new challenges. This study employs a
survey methodology to examine the policy landscape concerning these
technologies, drawing insights from 102 high school principals and higher
education provosts. Our results reveal a prominent policy gap: the majority of
institutions lack specialized guide-lines for the ethical deployment of AI
tools such as ChatGPT. Moreover,we observed that high schools are less inclined
to work on policies than higher educational institutions. Where such policies
do exist, they often overlook crucial issues, including student privacy and
algorithmic transparency. Administrators overwhelmingly recognize the necessity
of these policies, primarily to safeguard student safety and mitigate
plagiarism risks. Our findings underscore the urgent need for flexible and
iterative policy frameworks in educational contexts.

该研究通过调查方法检查了高中校长和高等教育主任们对促进人工智能等生成 AI 工具在教育环境中的伦理部署的政策现状，结果显示出存在重要政策空白、缺乏对学生隐私和算法透明度等关键问题的政策，并强调在教育背景下需要灵活迭代的政策框架。