We introduce Mirage, the first multi-level superoptimizer for tensor
programs. A key idea in Mirage is $\mu$Graphs, a uniform representation of
tensor programs at the kernel, thread block, and thread levels of the GPU
compute hierarchy. $\mu$Graphs enable Mirage to discover novel optimizations
that combine algebraic transformations, schedule transformations, and
generation of new custom kernels. To navigate the large search space, Mirage
introduces a pruning technique based on abstraction that significantly reduces
the search space and provides a certain optimality guarantee. To ensure that
the optimized $\mu$Graph is equivalent to the input program, Mirage introduces
a probabilistic equivalence verification procedure with strong theoretical
guarantees. Our evaluation shows that Mirage outperforms existing approaches by
up to 3.5$\times$ even for DNNs that are widely used and heavily optimized.
Mirage is publicly available at this https URL

Mirage 是首个多级超优化程序，通过引入一种称为 μGraphs 的统一表示方法，在 GPU 计算层次结构的内核、线程块和线程级别上对张量程序进行优化，包括代数转换、调度转换和生成新的自定义内核的新颖优化技术。通过抽象的剪枝技术和概率等价性验证过程，Mirage 在大大减少搜索空间的同时，确保优化的 μGraph 与输入程序等价，相比广泛使用且经过大量优化的 DNN，Mirage 的性能提升最高可达 3.5 倍。