One-shot coreset selection aims to select a representative subset of the
training data, given a pruning rate, that can later be used to train future
models while retaining high accuracy. State-of-the-art coreset selection
methods pick the highest importance examples based on an importance metric and
are found to perform well at low pruning rates. However, at high pruning rates,
they suffer from a catastrophic accuracy drop, performing worse than even
random sampling. This paper explores the reasons behind this accuracy drop both
theoretically and empirically. We first propose a novel metric to measure the
coverage of a dataset on a specific distribution by extending the classical
geometric set cover problem to a distribution cover problem. This metric helps
explain why coresets selected by SOTA methods at high pruning rates perform
poorly compared to random sampling because of worse data coverage. We then
propose a novel one-shot coreset selection method, Coverage-centric Coreset
Selection (CCS), that jointly considers overall data coverage upon a
distribution as well as the importance of each example. We evaluate CCS on five
datasets and show that, at high pruning rates (e.g., 90%), it achieves
significantly better accuracy than previous SOTA methods (e.g., at least 19.56%
higher on CIFAR10) as well as random selection (e.g., 7.04% higher on CIFAR10)
and comparable accuracy at low pruning rates. We make our code publicly
available at
this https URL.

本文研究一次性 coreset 选择，提出一种新的度量方法来度量训练数据集在特定分布下的覆盖率，将整体数据覆盖率与每个样本的重要性同时考虑，提出了 Coverage-centric Coreset Selection（CCS）方法，利用 CCS 在高精简率（例如 90%）时得到的精度显著高于之前的 SOTA 方法以及随机选择方法，并且在低精简率时具有类似的精度。

高裁剪率下基于覆盖率的核心集选择

Coverage-centric Coreset Selection for High Pruning Rates

Various forms of representations may arise in the many layers embedded in
deep neural networks (DNNs). Of these, where can we find the most compact
representation? We propose to use a pruning framework to answer this question:
How compact can each layer be compressed, without losing performance? Most of
the existing DNN compression methods do not consider the relative
compressibility of the individual layers. They uniformly apply a single target
sparsity to all layers or adapt layer sparsity using heuristics and additional
training. We propose a principled method that automatically determines the
sparsity of individual layers derived from the importance of each layer. To do
this, we consider a metric to measure the importance of each layer based on the
layer-wise capacity. Given the trained model and the total target sparsity, we
first evaluate the importance of each layer from the model. From the evaluated
importance, we compute the layer-wise sparsity of each layer. The proposed
method can be applied to any DNN architecture and can be combined with any
pruning method that takes the total target sparsity as a parameter. To validate
the proposed method, we carried out an image classification task with two types
of DNN architectures on two benchmark datasets and used three pruning methods
for compression. In case of VGG-16 model with weight pruning on the ImageNet
dataset, we achieved up to 75% (17.5% on average) better top-5 accuracy than
the baseline under the same total target sparsity. Furthermore, we analyzed
where the maximum compression can occur in the network. This kind of analysis
can help us identify the most compact representation within a deep neural
network.

本文提出了一种基于修剪框架的深度神经网络压缩方法，实现了自动确定各个层稀疏度的功能，并通过在两种 DNN 架构上进行图像分类任务的实验验证了该方法的有效性。