DataFlow has been emerging as a new paradigm for building task-oriented
chatbots due to its expressive semantic representations of the dialogue tasks.
Despite the availability of a large dataset SMCalFlow and a simplified syntax,
the development and evaluation of DataFlow-based chatbots remain challenging
due to the system complexity and the lack of downstream toolchains. In this
demonstration, we present DFEE, an interactive DataFlow Execution and
Evaluation toolkit that supports execution, visualization and benchmarking of
semantic parsers given dialogue input and backend database. We demonstrate the
system via a complex dialog task: event scheduling that involves temporal
reasoning. It also supports diagnosing the parsing results via a friendly
interface that allows developers to examine dynamic DataFlow and the
corresponding execution results. To illustrate how to benchmark SoTA models, we
propose a novel benchmark that covers more sophisticated event scheduling
scenarios and a new metric on task success evaluation. The codes of DFEE have
been released on this https URL

DFEE 是一个交互式数据流执行和评估工具包，支持语义解析器的执行、可视化和基准测试，提供友好的接口来诊断解析结果，还提出了一个更复杂的基准测试用例和任务成功评估指标来演示如何进行基准测试。