The emergence of instruction-tuned large language models (LLMs) has advanced the field of dialogue systems, enabling both realistic user simulations and robust multi-turn conversational agents. However, existing research often evaluates these components in isolation-either focusing on a single user simulator or a specific system design-limiting the generalisability of insights across architectures and configurations. In this work, we propose clem todd (chat-optimized LLMs for task-oriented dialogue systems development), a flexible framework for systematically evaluating dialogue systems under consistent conditions. clem todd enables detailed benchmarking across combinations of user simulators and dialogue systems, whether existing models from literature or newly developed ones. It supports plug-and-play integration and ensures uniform datasets, evaluation metrics, and computational constraints. We showcase clem todd's flexibility by re-evaluating existing task-oriented dialogue systems within this unified setup and integrating three newly proposed dialogue systems into the same evaluation pipeline. Our results provide actionable insights into how architecture, scale, and prompting strategies affect dialogue performance, offering practical guidance for building efficient and effective conversational AI systems.

本研究针对现有对话系统研究中缺乏系统性评估的问题，提出了一种灵活的框架 clem todd，用于在一致的条件下系统化评估对话系统。研究发现，该框架能够详细比较不同用户模拟器与对话系统的组合，提供关于架构、规模和提示策略对对话性能影响的可操作见解，为构建高效的会话人工智能系统提供实用指导。

clem:todd: 一种系统化基准测试基于LLM的任务导向对话系统实现的框架