Machine learning (ML) plays a pivotal role in detecting malicious software.
Despite the high F1-scores reported in numerous studies reaching upwards of
0.99, the issue is not completely solved. Malware detectors often experience
performance decay due to constantly evolving operating systems and attack
methods, which can render previously learned knowledge insufficient for
accurate decision-making on new inputs. This paper argues that commonly
reported results are inflated due to two pervasive sources of experimental bias
in the detection task: spatial bias caused by data distributions that are not
representative of a real-world deployment; and temporal bias caused by
incorrect time splits of data, leading to unrealistic configurations. To
address these biases, we introduce a set of constraints for fair experiment
design, and propose a new metric, AUT, for classifier robustness in real-world
settings. We additionally propose an algorithm designed to tune training data
to enhance classifier performance. Finally, we present TESSERACT, an
open-source framework for realistic classifier comparison. Our evaluation
encompasses both traditional ML and deep learning methods, examining published
works on an extensive Android dataset with 259,230 samples over a five-year
span. Additionally, we conduct case studies in the Windows PE and PDF domains.
Our findings identify the existence of biases in previous studies and reveal
that significant performance enhancements are possible through appropriate,
periodic tuning. We explore how mitigation strategies may support in achieving
a more stable and better performance over time by employing multiple strategies
to delay performance decay.

本论文通过引入公平实验设计的一组约束条件和 AUT 度量，提出了解决恶意软件检测任务中实验偏差的方法，并提供了一个能够增强分类器性能的算法，以及一个用于实际分类器比较的开源框架 TESSERACT。研究发现以前的研究存在偏差，并通过适时调整的多种策略来延缓性能下降，从而实现更稳定和更好的性能。

TESSERACT: 消除恶意软件分类中的实验偏见 —— 跨时空的研究（扩展版）

TESSERACT: Eliminating Experimental Bias in Malware Classification  across Space and Time (Extended Version)

Empirical design in reinforcement learning is no small task. Running good
experiments requires attention to detail and at times significant computational
resources. While compute resources available per dollar have continued to grow
rapidly, so have the scale of typical experiments in reinforcement learning. It
is now common to benchmark agents with millions of parameters against dozens of
tasks, each using the equivalent of 30 days of experience. The scale of these
experiments often conflict with the need for proper statistical evidence,
especially when comparing algorithms. Recent studies have highlighted how
popular algorithms are sensitive to hyper-parameter settings and implementation
details, and that common empirical practice leads to weak statistical evidence
(Machado et al., 2018; Henderson et al., 2018). Here we take this one step
further.
This manuscript represents both a call to action, and a comprehensive
resource for how to do good experiments in reinforcement learning. In
particular, we cover: the statistical assumptions underlying common performance
measures, how to properly characterize performance variation and stability,
hypothesis testing, special considerations for comparing multiple agents,
baseline and illustrative example construction, and how to deal with
hyper-parameters and experimenter bias. Throughout we highlight common mistakes
found in the literature and the statistical consequences of those in example
experiments. The objective of this document is to provide answers on how we can
use our unprecedented compute to do good science in reinforcement learning, as
well as stay alert to potential pitfalls in our empirical design.

本文旨在提出在强化学习中进行良好实验的方法，并强调常见错误和潜在统计结果，覆盖了如何对性能进行妥善表征、假设检验、比较多个代理、基准和说明性例子的构建、如何处理超参数和实验偏差等等，旨在通过充分利用计算资源来进行良好的实证研究。

强化学习中的经验设计

Empirical Design in Reinforcement Learning

Is Android malware classification a solved problem? Published F1 scores of up
to 0.99 appear to leave very little room for improvement. In this paper, we
argue that results are commonly inflated due to two pervasive sources of
experimental bias: "spatial bias" caused by distributions of training and
testing data that are not representative of a real-world deployment; and
"temporal bias" caused by incorrect time splits of training and testing sets,
leading to impossible configurations. We propose a set of space and time
constraints for experiment design that eliminates both sources of bias. We
introduce a new metric that summarizes the expected robustness of a classifier
in a real-world setting, and we present an algorithm to tune its performance.
Finally, we demonstrate how this allows us to evaluate mitigation strategies
for time decay such as active learning. We have implemented our solutions in
TESSERACT, an open source evaluation framework for comparing malware
classifiers in a realistic setting. We used TESSERACT to evaluate three Android
malware classifiers from the literature on a dataset of 129K applications
spanning over three years. Our evaluation confirms that earlier published
results are biased, while also revealing counter-intuitive performance and
showing that appropriate tuning can lead to significant improvements.

本文提出一种针对 Android 恶意软件分类的实验设计约束集，解决了空间偏差和时间偏差造成的指标过度乐观的问题，并提出了一种新的分类器鲁棒性的总结度量标准并进行了性能调整。最后，通过对 TESSERACT 实现的三个 Android 恶意软件分类器进行数据集实验，证实了之前的结果出现的偏差以及准确性提高需要适当的调整。