In recent years there has been a shift from heuristics-based malware
detection towards machine learning, which proves to be more robust in the
current heavily adversarial threat landscape. While we acknowledge machine
learning to be better equipped to mine for patterns in the increasingly high
amounts of similar-looking files, we also note a remarkable scarcity of the
data available for similarity-targeted research. Moreover, we observe that the
focus in the few related works falls on quantifying similarity in malware,
often overlooking the clean data. This one-sided quantification is especially
dangerous in the context of detection bypass. We propose to address the
deficiencies in the space of similarity research on binary files, starting from
EMBER - one of the largest malware classification data sets. We enhance EMBER
with similarity information as well as malware class tags, to enable further
research in the similarity space. Our contribution is threefold: (1) we publish
EMBERSim, an augmented version of EMBER, that includes similarity-informed
tags; (2) we enrich EMBERSim with automatically determined malware class tags
using the open-source tool AVClass on VirusTotal data and (3) we describe and
share the implementation for our class scoring technique and leaf similarity
method.

近年来，从基于启发式的恶意软件检测转向机器学习，后者对于当前存在严峻的对抗性威胁环境更为强大。然而，我们注意到相似性研究可用数据稀缺，导致相关作品偏重量化恶意软件的相似性而忽视了干净数据，这种单方量化在检测绕过中尤其危险。为此，我们提出从二进制文件的相似性研究空间入手，解决这些不足，并以大型恶意软件分类数据集 EMBER 为基础进行增强，加入相似性信息以及恶意软件类别标签以促进进一步的相似性研究。我们的贡献是三方面的：(1) 发布了 EMBERSim，EMBER 的增强版本，包括相似性提供的标签；(2) 使用开源工具 AVClass 在 VirusTotal 数据上自动确定恶意软件类别标签，丰富了 EMBERSim；(3) 描述并分享了我们的类别评分技术和叶相似性方法的实现。

EMBERSim: 用于提升恶意软件分析中相似性搜索的大规模数据库

EMBERSim: A Large-Scale Databank for Boosting Similarity Search in  Malware Analysis

Malware remains a serious problem for corporations, government agencies, and
individuals, as attackers continue to use it as a tool to effect frequent and
costly network intrusions. Machine learning holds the promise of automating the
work required to detect newly discovered malware families, and could
potentially learn generalizations about malware and benign software that
support the detection of entirely new, unknown malware families. Unfortunately,
few proposed machine learning based malware detection methods have achieved the
low false positive rates required to deliver deployable detectors.
In this paper we a deep neural network malware classifier that achieves a
usable detection rate at an extremely low false positive rate and scales to
real world training example volumes on commodity hardware. Specifically, we
show that our system achieves a 95% detection rate at 0.1% false positive rate
(FPR), based on more than 400,000 software binaries sourced directly from our
customers and internal malware databases. We achieve these results by directly
learning on all binaries, without any filtering, unpacking, or manually
separating binary files into categories. Further, we confirm our false positive
rates directly on a live stream of files coming in from Invincea's deployed
endpoint solution, provide an estimate of how many new binary files we expected
to see a day on an enterprise network, and describe how that relates to the
false positive rate and translates into an intuitive threat score.
Our results demonstrate that it is now feasible to quickly train and deploy a
low resource, highly accurate machine learning classification model, with false
positive rates that approach traditional labor intensive signature based
methods, while also detecting previously unseen malware.

本文介绍了一种基于深度神经网络的恶意软件分类器，可以在商品硬件上处理实际金融企业数据达到 95% 的检测率和 0.1% 的误报率。