As the number of pre-trained machine learning (ML) models is growing
exponentially, data reduction tools are not catching up. Existing data
reduction techniques are not specifically designed for pre-trained model (PTM)
dataset files. This is largely due to a lack of understanding of the patterns
and characteristics of these datasets, especially those relevant to data
reduction and compressibility.
This paper presents the first, exhaustive analysis to date of PTM datasets on
storage compressibility. Our analysis spans different types of data reduction
and compression techniques, from hash-based data deduplication, data similarity
detection, to dictionary-coding compression. Our analysis explores these
techniques at three data granularity levels, from model layers, model chunks,
to model parameters. We draw new observations that indicate that modern data
reduction tools are not effective when handling PTM datasets. There is a
pressing need for new compression methods that take into account PTMs' data
characteristics for effective storage reduction.
Motivated by our findings, we design ELF, a simple yet effective,
error-bounded, lossy floating-point compression method. ELF transforms
floating-point parameters in such a way that the common exponent field of the
transformed parameters can be completely eliminated to save storage space. We
develop Elves, a compression framework that integrates ELF along with several
other data reduction methods. Elves uses the most effective method to compress
PTMs that exhibit different patterns. Evaluation shows that Elves achieves an
overall compression ratio of $1.52\times$, which is $1.31\times$, $1.32\times$
and $1.29\times$ higher than a general-purpose compressor (zstd), an
error-bounded lossy compressor (SZ3), and the uniform model quantization,
respectively, with negligible model accuracy loss.

论文通过对预训练模型数据集进行详尽分析，揭示了现代数据压缩工具对预训练模型数据集不够有效的问题，并提出了一种简单而有效的浮点数压缩方法及与其他数据减少方法相结合的压缩框架。