This work proposes a novel method to generate C-Tests; a deviated form of
cloze tests (a gap filling exercise) where only the last part of a word is
turned into a gap. In contrast to previous works that only consider varying the
gap size or gap placement to achieve locally optimal solutions, we propose a
mixed-integer programming (MIP) approach. This allows us to consider gap size
and placement simultaneously, achieving globally optimal solutions, and to
directly integrate state-of-the-art models for gap difficulty prediction into
the optimization problem. A user study with 40 participants across four C-Test
generation strategies (including GPT-4) shows that our approach (MIP)
significantly outperforms two of the baseline strategies (based on gap
placement and GPT-4); and performs on-par with the third (based on gap size).
Our analysis shows that GPT-4 still struggles to fulfill explicit constraints
during generation and that MIP produces C-Tests that correlate best with the
perceived difficulty. We publish our code, model, and collected data consisting
of 32 English C-Tests with 20 gaps each (totaling 3,200 individual gap
responses) under an open source license.

该研究论文提出了一种新的方法来生成 C-Tests，它是一种 cloze 测试的变体，在这里只有单词的最后一部分变成了间隙。与以往只考虑变化间隙大小或间隙位置以获得局部最优解的方法相比，我们提出了一种混合整数规划（MIP）方法。这使我们能够同时考虑间隙大小和位置，实现全局最优解，并直接将最先进的间隙难度预测模型集成到优化问题中。通过与四种 C-Test 生成策略（包括 GPT-4）的 40 名参与者的用户研究，我们发现我们的方法（MIP）明显优于两种基准策略（基于间隙位置和 GPT-4），并且与第三种策略（基于间隙大小）表现相当。我们的分析表明，GPT-4 在生成过程中仍然难以满足显式约束条件，而 MIP 生成的 C-Tests 与所感知的困难程度最相关。我们在开源许可下发布了我们的代码、模型和收集的数据，其中包括 32 个英语 C-Tests，每个测试包含 20 个间隙（总共 3200 个单独的间隙响应）。

基于混合整数规划的约束 C - 测试生成

Constrained C-Test Generation via Mixed-Integer Programming

With the increased adoption of artificial intelligence (AI) in industry and
society, effective human-AI interaction systems are becoming increasingly
important. A central challenge in the interaction of humans with AI is the
estimation of difficulty for human and AI agents for single task
instances.These estimations are crucial to evaluate each agent's capabilities
and, thus, required to facilitate effective collaboration. So far, research in
the field of human-AI interaction estimates the perceived difficulty of humans
and AI independently from each other. However, the effective interaction of
human and AI agents depends on metrics that accurately reflect each agent's
perceived difficulty in achieving valuable outcomes. Research to date has not
yet adequately examined the differences in the perceived difficulty of humans
and AI. Thus, this work reviews recent research on the perceived difficulty in
human-AI interaction and contributing factors to consistently compare each
agent's perceived difficulty, e.g., creating the same prerequisites.
Furthermore, we present an experimental design to thoroughly examine the
perceived difficulty of both agents and contribute to a better understanding of
the design of such systems.

本文研究了人工智能与人类交互中的感知难度，探讨了如何确切地评估每个代理人的能力，并提出了一种实验设计，以全面检验两个代理人的感知难度，为更好地设计这样的系统做出贡献。