Building and maintaining large AI fleets to efficiently support the
fast-growing DL workloads is an active research topic for modern cloud
infrastructure providers. Generating accurate benchmarks plays an essential
role in the design and evaluation of rapidly evoloving software and hardware
solutions in this area. Two fundamental challenges to make this process
scalable are (i) workload representativeness and (ii) the ability to quickly
incorporate changes to the fleet into the benchmarks.
To overcome these issues, we propose Mystique, an accurate and scalable
framework for production AI benchmark generation. It leverages the PyTorch
execution graph (EG), a new feature that captures the runtime information of AI
models at the granularity of operators, in a graph format, together with their
metadata. By sourcing EG traces from the fleet, we can build AI benchmarks that
are portable and representative. Mystique is scalable, with its lightweight
data collection, in terms of runtime overhead and user instrumentation efforts.
It is also adaptive, as the expressiveness and composability of EG format
allows flexible user control over benchmark creation.
We evaluate our methodology on several production AI workloads, and show that
benchmarks generated with Mystique closely resemble original AI models, both in
execution time and system-level metrics. We also showcase the portability of
the generated benchmarks across platforms, and demonstrate several use cases
enabled by the fine-grained composability of the execution graph.

该论文提出了 Mystique 框架，利用 PyTorch execution graph 进行轻量级的数据收集和灵活的基于运行时信息的 AI benchmark 生成，使生成的 AI benchmarks 在表示性和可移植性上都得到了很好的评价。