This study introduces an innovative approach aimed at the efficient pruning
of neural networks, with a particular focus on their deployment on edge
devices. Our method involves the integration of the Lottery Ticket Hypothesis
(LTH) with the Knowledge Distillation (KD) framework, resulting in the
formulation of three distinct pruning models. These models have been developed
to address scalability issue in recommender systems, whereby the complexities
of deep learning models have hindered their practical deployment. With
judicious application of the pruning techniques, we effectively curtail the
power consumption and model dimensions without compromising on accuracy.
Empirical evaluation has been performed using two real world datasets from
diverse domains against two baselines. Gratifyingly, our approaches yielded a
GPU computation-power reduction of up to 66.67%. Notably, our study contributes
to the field of recommendation system by pioneering the application of LTH and
KD.

该研究介绍了一种创新方法，旨在高效剪枝神经网络，特别关注其在边缘设备上的部署。通过将 “Lottery Ticket Hypothesis”（LTH）与 “Knowledge Distillation”（KD）框架相结合，我们提出了三个不同的剪枝模型，用于解决推荐系统中的可扩展性问题。通过巧妙应用剪枝技术，我们有效地降低了功耗和模型维度，同时又不影响准确性。实证评估使用来自不同领域的两个真实数据集对比了两个基准方法。令人满意的是，我们的方法在 GPU 计算能力上实现了高达 66.67% 的降低。值得注意的是，我们的研究通过首次应用 LTH 和 KD 在推荐系统领域做出了贡献。

通过彩票票据假设和基于知识蒸馏的神经网络修剪增强推荐系统的可扩展性

Enhancing Scalability in Recommender Systems through Lottery Ticket  Hypothesis and Knowledge Distillation-based Neural Network Pruning

Overparameterized transformer networks have obtained state of the art results
in various natural language processing tasks, such as machine translation,
language modeling, and question answering. These models contain hundreds of
millions of parameters, necessitating a large amount of computation and making
them prone to overfitting. In this work, we explore LayerDrop, a form of
structured dropout, which has a regularization effect during training and
allows for efficient pruning at inference time. In particular, we show that it
is possible to select sub-networks of any depth from one large network without
having to finetune them and with limited impact on performance. We demonstrate
the effectiveness of our approach by improving the state of the art on machine
translation, language modeling, summarization, question answering, and language
understanding benchmarks. Moreover, we show that our approach leads to small
BERT-like models of higher quality compared to training from scratch or using
distillation.

本文介绍了一种新的结构化 dropout 形式 ——LayerDrop，该形式可在训练过程中实现正则化效果，在推理时允许有效的剪枝。作者通过实验证明了该方法对机器翻译、语言模型、文本摘要、问答和语言理解等基准测试的提升，并提出使用该方法可以得到更高质量的类 BERT 模型。