We present a conceptual framework that unifies a variety of evaluation
metrics for different structured prediction tasks (e.g. event and relation
extraction, syntactic and semantic parsing). Our framework requires
representing the outputs of these tasks as objects of certain data types, and
derives metrics through matching of common substructures, possibly followed by
normalization. We demonstrate how commonly used metrics for a number of tasks
can be succinctly expressed by this framework, and show that new metrics can be
naturally derived in a bottom-up way based on an output structure. We release a
library that enables this derivation to create new metrics. Finally, we
consider how specific characteristics of tasks motivate metric design
decisions, and suggest possible modifications to existing metrics in line with
those motivations.

我们提出一个概念框架，统一了不同结构预测任务（例如事件和关系提取，句法和语义解析）的各种评估指标。通过将这些任务的输出表示为某些数据类型的对象，并通过匹配共同子结构，可能跟随标准化，推导出指标。我们演示了如何用这个框架简洁地表达许多任务的常用指标，并显示可以自然地基于输出结构自下而上地推导出新的指标。我们发布了一个库，使得可以派生出新的指标。最后，我们考虑了任务的特定特性如何推动指标设计决策，并建议根据这些动机可能修改现有的指标。

结构化预测的评估度量的统一视角

A Unified View of Evaluation Metrics for Structured Prediction

Natural language (NL) feedback contains rich information about the user
experience. Existing studies focus on an instance-level approach, where
feedback is used to refine specific examples, disregarding its system-wide
application. This paper proposes a general framework for unlocking the
system-level use of NL feedback. We show how to use feedback to formalize
system-level design decisions in a human-in-the-loop-process -- in order to
produce better models. In particular this is done through: (i) metric design
for tasks; and (ii) language model prompt design for refining model responses.
We conduct two case studies of this approach for improving search query
generation and dialog response generation, demonstrating the effectiveness of
the use of system-level feedback. We show the combination of system-level
feedback and instance-level feedback brings further gains, and that human
written instance-level feedback results in more grounded refinements than
GPT-3.5 written ones, underlying the importance of human feedback for building
systems.

本文提出了一种通用框架，利用自然语言反馈来解锁系统级反馈的用处，并应用于搜索和对话系统的查询和响应生成中，本方法证明了系统级反馈与实例级反馈的结合带来了更多的收益，并强调了人类反馈对系统构建的重要性。