Posing reading comprehension as a generation problem provides a great deal of
flexibility, allowing for open-ended questions with few restrictions on
possible answers. However, progress is impeded by existing generation metrics,
which rely on token overlap and are agnostic to the nuances of reading
comprehension. To address this, we introduce a benchmark for training and
evaluating generative reading comprehension metrics: MOdeling Correctness with
Human Annotations. MOCHA contains 40K human judgement scores on model outputs
from 6 diverse question answering datasets and an additional set of minimal
pairs for evaluation. Using MOCHA, we train a Learned Evaluation metric for
Reading Comprehension, LERC, to mimic human judgement scores. LERC outperforms
baseline metrics by 10 to 36 absolute Pearson points on held-out annotations.
When we evaluate robustness on minimal pairs, LERC achieves 80% accuracy,
outperforming baselines by 14 to 26 absolute percentage points while leaving
significant room for improvement. MOCHA presents a challenging problem for
developing accurate and robust generative reading comprehension metrics.

将阅读理解视为生成问题可提供更多的灵活性，但现有的生成度量依赖于词汇重叠，对于阅读理解的细微 nuance 不偏不倚。为解决这个问题，我们引入了 MOCHA 作为评估阅读理解模型度量的基准，并使用 LERC 作为训练模型来模仿人类判断分数。使用 MOCHA 数据集，LERC 的表现优于基准度量，并在最小配对的鲁棒性评估上达到了 80％的准确度，这个难题为开发准确且鲁棒的生成阅读理解度量提供了重要参考。

MOCHA：用于训练和评估生成式阅读理解指标的数据集

MOCHA: A Dataset for Training and Evaluating Generative Reading  Comprehension Metrics

Federated learning poses new statistical and systems challenges in training
machine learning models over distributed networks of devices. In this work, we
show that multi-task learning is naturally suited to handle the statistical
challenges of this setting, and propose a novel systems-aware optimization
method, MOCHA, that is robust to practical systems issues. Our method and
theory for the first time consider issues of high communication cost,
stragglers, and fault tolerance for distributed multi-task learning. The
resulting method achieves significant speedups compared to alternatives in the
federated setting, as we demonstrate through simulations on real-world
federated datasets.

本文研究了在分布式设备网络上训练机器学习模型所面临的统计和系统挑战，提出了一种适用于多任务学习的优化方法 MOCHA，并在考虑通信成本、慢设备和容错性等实际系统问题的前提下，实现了比替代方法更显著的加速效果。