Human perceptual studies are the gold standard for the evaluation of many
research tasks in machine learning, linguistics, and psychology. However, these
studies require significant time and cost to perform. As a result, many
researchers use objective measures that can correlate poorly with human
evaluation. When subjective evaluations are performed, they are often not
reported with sufficient detail to ensure reproducibility. We propose
Reproducible Subjective Evaluation (ReSEval), an open-source framework for
quickly deploying crowdsourced subjective evaluations directly from Python.
ReSEval lets researchers launch A/B, ABX, Mean Opinion Score (MOS) and MUltiple
Stimuli with Hidden Reference and Anchor (MUSHRA) tests on audio, image, text,
or video data from a command-line interface or using one line of Python, making
it as easy to run as objective evaluation. With ReSEval, researchers can
reproduce each other's subjective evaluations by sharing a configuration file
and the audio, image, text, or video files.

本文提出了 ReSEval，这是一种开源框架，可从 Python 直接快速部署 crowdsourced 主观评估，以方便研究人员运行主观评估以及通过共享配置文件和音频、图像、文本或视频文件来重现彼此的主观评估。

可重现的主观评估

Reproducible Subjective Evaluation

We introduce a large scale crowdsourced text adventure game as a research
platform for studying grounded dialogue. In it, agents can perceive, emote, and
act whilst conducting dialogue with other agents. Models and humans can both
act as characters within the game. We describe the results of training
state-of-the-art generative and retrieval models in this setting. We show that
in addition to using past dialogue, these models are able to effectively use
the state of the underlying world to condition their predictions. In
particular, we show that grounding on the details of the local environment,
including location descriptions, and the objects (and their affordances) and
characters (and their previous actions) present within it allows better
predictions of agent behavior and dialogue. We analyze the ingredients
necessary for successful grounding in this setting, and how each of these
factors relate to agents that can talk and act successfully.

本文介绍了一个大规模的众包文字冒险游戏作为研究平台，用于研究基于场景的对话。在其中，代理可以感知，表情和行动，同时与其他代理进行对话。我们介绍了在这种情况下训练最先进的生成模型和检索模型的结果。我们展示了除了过去的对话，这些模型还能够有效地使用潜在世界的状态来调整它们的预测，特别是使用本地环境的详细信息，包括位置描述和其中存在的对象（及其可用性）和角色（及其先前的行动），可以更好地预测代理行为和对话。我们分析了在这种设置中成功落地所需的要素，以及每个因素如何与能够成功说话和行动的代理相关。