In this paper, we assess the viability of transformer models in end-to-end
InfoSec settings, in which no intermediate feature representations or
processing steps occur outside the model. We implement transformer models for
two distinct InfoSec data formats - specifically URLs and PE files - in a novel
end-to-end approach, and explore a variety of architectural designs, training
regimes, and experimental settings to determine the ingredients necessary for
performant detection models. We show that in contrast to conventional
transformers trained on more standard NLP-related tasks, our URL transformer
model requires a different training approach to reach high performance levels.
Specifically, we show that 1) pre-training on a massive corpus of unlabeled URL
data for an auto-regressive task does not readily transfer to binary
classification of malicious or benign URLs, but 2) that using an auxiliary
auto-regressive loss improves performance when training from scratch. We
introduce a method for mixed objective optimization, which dynamically balances
contributions from both loss terms so that neither one of them dominates. We
show that this method yields quantitative evaluation metrics comparable to that
of several top-performing benchmark classifiers. Unlike URLs, binary
executables contain longer and more distributed sequences of information-rich
bytes. To accommodate such lengthy byte sequences, we introduce additional
context length into the transformer by providing its self-attention layers with
an adaptive span similar to Sukhbaatar et al. We demonstrate that this approach
performs comparably to well-established malware detection models on benchmark
PE file datasets, but also point out the need for further exploration into
model improvements in scalability and compute efficiency.

本研究评估 transformer 模型在端到端 InfoSec 设置中的可行性，并探索了在两种不同的 InfoSec 数据格式（即 URL 和 PE 文件）上实施 transformer 模型的方法，以及多种不同的架构设计、训练和实验设置以确定实现性能检测模型所需的条件，通过实验证明与传统的 transformers 相比，URL 转换器模型需要不同的训练方法才能达到高性能水平。