When investigating a malicious file, searching for related files is a common
task that malware analysts must perform. Given that production malware corpora
may contain over a billion files and consume petabytes of storage, many feature
extraction and similarity search approaches are computationally infeasible. Our
work explores the potential of antivirus (AV) scan data as a scalable source of
features for malware. This is possible because AV scan reports are widely
available through services such as VirusTotal and are ~100x smaller than the
average malware sample. The information within an AV scan report is abundant
with information and can indicate a malicious file's family, behavior, target
operating system, and many other characteristics. We introduce AVScan2Vec, a
language model trained to comprehend the semantics of AV scan data. AVScan2Vec
ingests AV scan data for a malicious file and outputs a meaningful vector
representation. AVScan2Vec vectors are ~3 to 85x smaller than popular
alternatives in use today, enabling faster vector comparisons and lower memory
usage. By incorporating Dynamic Continuous Indexing, we show that
nearest-neighbor queries on AVScan2Vec vectors can scale to even the largest
malware production datasets. We also demonstrate that AVScan2Vec vectors are
superior to other leading malware feature vector representations across nearly
all classification, clustering, and nearest-neighbor lookup algorithms that we
evaluated.

本文利用杀毒软件扫描数据作为特征来进行恶意软件文件的相关性查询，并引入 AVScan2Vec 语言模型用以表示数据特征向量，结合 DCI 方法，实现了 AVScan2Vec 向量上的最近邻查询，结果显示 AVScan2Vec 向量表示方式优于其他恶意软件特征表示方式。

AVScan2Vec: 基于杀毒扫描数据的特征学习，应用于大规模恶意软件语料库

AVScan2Vec: Feature Learning on Antivirus Scan Data for Production-Scale  Malware Corpora

Most exact methods for k-nearest neighbour search suffer from the curse of
dimensionality; that is, their query times exhibit exponential dependence on
either the ambient or the intrinsic dimensionality. Dynamic Continuous Indexing
(DCI) offers a promising way of circumventing the curse and successfully
reduces the dependence of query time on intrinsic dimensionality from
exponential to sublinear. In this paper, we propose a variant of DCI, which we
call Prioritized DCI, and show a remarkable improvement in the dependence of
query time on intrinsic dimensionality. In particular, a linear increase in
intrinsic dimensionality, or equivalently, an exponential increase in the
number of points near a query, can be mostly counteracted with just a linear
increase in space. We also demonstrate empirically that Prioritized DCI
significantly outperforms prior methods. In particular, relative to
Locality-Sensitive Hashing (LSH), Prioritized DCI reduces the number of
distance evaluations by a factor of 14 to 116 and the memory consumption by a
factor of 21.

该研究介绍了一种名为优先的动态连续索引（Prioritized DCI）的变体，用于 k 近邻搜索，并且相对于现有方法（如局部敏感哈希，LSH），优先 DCI 通过线性增加空间而不是查询时间的依赖来解决了维数灾难的问题，并在内在维数方面展现了显着的改进。