We introduce BotSIM, a modular, open-source Bot SIMulation environment with
dialog generation, user simulation and conversation analytics capabilities.
BotSIM aims to serve as a one-stop solution for large-scale data-efficient
end-to-end evaluation, diagnosis and remediation of commercial task-oriented
dialog (TOD) systems to significantly accelerate commercial bot development and
evaluation, reduce cost and time-to-market. BotSIM adopts a layered design
comprising the infrastructure layer, the adaptor layer and the application
layer. The infrastructure layer hosts key models and components to support
BotSIM's major functionalities via a streamlined
"generation-simulation-remediation" pipeline. The adaptor layer is used to
extend BotSIM to accommodate new bot platforms. The application layer provides
a suite of command line tools and a Web App to significantly lower the entry
barrier for BotSIM users such as bot admins or practitioners. In this report,
we focus on the technical designs of various system components. A detailed case
study using Einstein BotBuilder is also presented to show how to apply BotSIM
pipeline for bot evaluation and remediation. The detailed system descriptions
can be found in our system demo paper. The toolkit is available at:
this https URL .

BotSIM 是一个模块化、开源的机器人仿真环境，具有对话生成、用户模拟和对话分析功能，旨在成为大规模、数据高效的商业任务导向对话系统评估、诊断和修复的一站式解决方案，以显著加快商业机器人的开发和评估，降低成本和上市时间。

BotSIM：商业任务导向对话系统的端到端对话机器人模拟工具包

BotSIM: An End-to-End Bot Simulation Toolkit for Commercial Task-Oriented Dialog Systems

We present BotSIM, a data-efficient end-to-end Bot SIMulation toolkit for
commercial text-based task-oriented dialog (TOD) systems. BotSIM consists of
three major components: 1) a Generator that can infer semantic-level dialog
acts and entities from bot definitions and generate user queries via
model-based paraphrasing; 2) an agenda-based dialog user Simulator (ABUS) to
simulate conversations with the dialog agents; 3) a Remediator to analyze the
simulated conversations, visualize the bot health reports and provide
actionable remediation suggestions for bot troubleshooting and improvement. We
demonstrate BotSIM's effectiveness in end-to-end evaluation, remediation and
multi-intent dialog generation via case studies on two commercial bot
platforms. BotSIM's "generation-simulation-remediation" paradigm accelerates
the end-to-end bot evaluation and iteration process by: 1) reducing manual test
cases creation efforts; 2) enabling a holistic gauge of the bot in terms of NLU
and end-to-end performance via extensive dialog simulation; 3) improving the
bot troubleshooting process with actionable suggestions. A demo of our system
can be found at this https URL and a demo video at
this https URL. We have open-sourced the toolkit at
this https URL

BotSIM 是一种数据高效的端到端 Bot 模拟工具包，用于商业文本任务导向对话系统，其包括生成器、基于议程的对话用户模拟器和修复程序，加速端到端 Bot 评估和迭代过程。