Handling the ever-increasing scale of contemporary deep learning and
transformer-based models poses a significant challenge. Although great strides
have been made in optimizing model compression techniques such as model
architecture search and knowledge distillation, the availability of data and
computational resources remains a considerable hurdle for these optimizations.
This paper introduces LayerCollapse, a novel alternative adaptive model
compression methodology. LayerCollapse works by eliminating non-linearities
within the network and collapsing two consecutive fully connected layers into a
single linear transformation. This approach simultaneously reduces both the
number of layers and the parameter count, thereby enhancing model efficiency.
We also introduce a compression aware regularizer, which compresses the model
in alignment with the dataset quality and model expressiveness, consequently
reducing overfitting across tasks. Our results demonstrate LayerCollapse's
effective compression and regularization capabilities in multiple fine-grained
classification benchmarks, achieving up to 74% post training compression with
minimal accuracy loss. We compare this method with knowledge distillation on
the same target network, showcasing a five-fold increase in computational
efficiency and 8% improvement in overall accuracy on the ImageNet dataset.

利用 LayerCollapse 方法进行适应性模型压缩，通过消除网络中的非线性并将两个连续的全连接层合并为单个线性变换，同时减少层数和参数数量，提高模型效率；引入压缩感知正则化器，根据数据集质量和模型表现压缩模型，减少过拟合；实验证明 LayerCollapse 在多个细粒度分类基准测试中实现有效的压缩和正则化，最高在训练后压缩 74% 但准确度损失最小；与知识蒸馏方法对比，在 ImageNet 数据集上计算效率提高五倍，整体准确度提高 8%。