Industry practitioners care about small improvements in malware detection
accuracy because their models are deployed to hundreds of millions of machines,
meaning a 0.1\% change can cause an overwhelming number of false positives.
However, academic research is often restrained to public datasets on the order
of ten thousand samples and is too small to detect improvements that may be
relevant to industry. Working within these constraints, we devise an approach
to generate a benchmark of configurable difficulty from a pool of available
samples. This is done by leveraging malware family information from tools like
AVClass to construct training/test splits that have different generalization
rates, as measured by a secondary model. Our experiments will demonstrate that
using a less accurate secondary model with disparate features is effective at
producing benchmarks for a more sophisticated target model that is under
evaluation. We also ablate against alternative designs to show the need for our
approach.

利用现有样本池，从可配置的难度方面生成基准，通过使用具有不同特征的不太准确的次级模型，针对正在评估的更复杂目标模型，有效地生成基准，以期改善恶意软件检测准确性。

恶意软件检测的小效果大小？加大训练 / 测试分割！

Small Effect Sizes in Malware Detection? Make Harder Train/Test Splits!

The exponential increase of hardware-software complexity has made it
impossible for compiler engineers to find the right optimization heuristics
manually. Predictive models have been shown to find near optimal heuristics
with little human effort but they are limited by a severe lack of diverse
benchmarks to train on. Generative AI has been used by researchers to
synthesize benchmarks into existing datasets. However, the synthetic programs
are short, exceedingly simple and lacking diversity in their features.
We develop BenchPress, the first ML compiler benchmark generator that can be
directed within source code feature representations. BenchPress synthesizes
executable functions by infilling code that conditions on the program's left
and right context. BenchPress uses active learning to introduce new benchmarks
with unseen features into the dataset of Grewe's et al. CPU vs GPU heuristic,
improving its acquired performance by 50%. BenchPress targets features that has
been impossible for other synthesizers to reach. In 3 feature spaces, we
outperform human-written code from GitHub, CLgen, CLSmith and the SRCIROR
mutator in targeting the features of Rodinia benchmarks.
BenchPress steers generation with beam search over a feature-agnostic
language model. We improve this with BenchDirect which utilizes a directed LM
that infills programs by jointly observing source code context and the compiler
features that are targeted. BenchDirect achieves up to 36% better accuracy in
targeting the features of Rodinia benchmarks, it is 1.8x more likely to give an
exact match and it speeds up execution time by up to 72% compared to
BenchPress. Both our models produce code that is difficult to distinguish from
human-written code. We conduct a Turing test which shows our models' synthetic
benchmarks are labelled as 'human-written' as often as human-written code from
GitHub.

本文提出了 BenchPress 和 BenchDirect，利用机器学习编译器生成基准测试案例，能够在源代码特征表示内定向生成可执行的函数，通过主动学习向已有数据集中引入具有未见过特征的新基准测试案例，成功地针对 Rodinia 的基准测试的特征进行了生成，模型的辨别和准确度效果优于现有方法。

BenchDirect: 用于编译器基准测试的定向语言模型

BenchDirect: A Directed Language Model for Compiler Benchmarks

Building and maintaining large AI fleets to efficiently support the
fast-growing DL workloads is an active research topic for modern cloud
infrastructure providers. Generating accurate benchmarks plays an essential
role in the design and evaluation of rapidly evoloving software and hardware
solutions in this area. Two fundamental challenges to make this process
scalable are (i) workload representativeness and (ii) the ability to quickly
incorporate changes to the fleet into the benchmarks.
To overcome these issues, we propose Mystique, an accurate and scalable
framework for production AI benchmark generation. It leverages the PyTorch
execution graph (EG), a new feature that captures the runtime information of AI
models at the granularity of operators, in a graph format, together with their
metadata. By sourcing EG traces from the fleet, we can build AI benchmarks that
are portable and representative. Mystique is scalable, with its lightweight
data collection, in terms of runtime overhead and user instrumentation efforts.
It is also adaptive, as the expressiveness and composability of EG format
allows flexible user control over benchmark creation.
We evaluate our methodology on several production AI workloads, and show that
benchmarks generated with Mystique closely resemble original AI models, both in
execution time and system-level metrics. We also showcase the portability of
the generated benchmarks across platforms, and demonstrate several use cases
enabled by the fine-grained composability of the execution graph.

该论文提出了 Mystique 框架，利用 PyTorch execution graph 进行轻量级的数据收集和灵活的基于运行时信息的 AI benchmark 生成，使生成的 AI benchmarks 在表示性和可移植性上都得到了很好的评价。