The exponential increase of hardware-software complexity has made it
impossible for compiler engineers to find the right optimization heuristics
manually. Predictive models have been shown to find near optimal heuristics
with little human effort but they are limited by a severe lack of diverse
benchmarks to train on. Generative AI has been used by researchers to
synthesize benchmarks into existing datasets. However, the synthetic programs
are short, exceedingly simple and lacking diversity in their features.
We develop BenchPress, the first ML compiler benchmark generator that can be
directed within source code feature representations. BenchPress synthesizes
executable functions by infilling code that conditions on the program's left
and right context. BenchPress uses active learning to introduce new benchmarks
with unseen features into the dataset of Grewe's et al. CPU vs GPU heuristic,
improving its acquired performance by 50%. BenchPress targets features that has
been impossible for other synthesizers to reach. In 3 feature spaces, we
outperform human-written code from GitHub, CLgen, CLSmith and the SRCIROR
mutator in targeting the features of Rodinia benchmarks.
BenchPress steers generation with beam search over a feature-agnostic
language model. We improve this with BenchDirect which utilizes a directed LM
that infills programs by jointly observing source code context and the compiler
features that are targeted. BenchDirect achieves up to 36% better accuracy in
targeting the features of Rodinia benchmarks, it is 1.8x more likely to give an
exact match and it speeds up execution time by up to 72% compared to
BenchPress. Both our models produce code that is difficult to distinguish from
human-written code. We conduct a Turing test which shows our models' synthetic
benchmarks are labelled as 'human-written' as often as human-written code from
GitHub.

本文提出了 BenchPress 和 BenchDirect，利用机器学习编译器生成基准测试案例，能够在源代码特征表示内定向生成可执行的函数，通过主动学习向已有数据集中引入具有未见过特征的新基准测试案例，成功地针对 Rodinia 的基准测试的特征进行了生成，模型的辨别和准确度效果优于现有方法。