Social media platforms prevent malicious activities by detecting harmful content of posts and comments. To that end, they employ large-scale deep neural network language models for sentiment analysis and content understanding. Some models, like BERT, are complex, and have numerous parameters, which makes them expensive to operate and maintain. To overcome these deficiencies, industry experts employ a knowledge distillation compression technique, where a distilled model is trained to reproduce the classification behavior of the original model. The distillation processes terminates when the distillation loss function reaches the stopping criteria. This function is mainly designed to ensure that the original and the distilled models exhibit alike classification behaviors. However, besides classification accuracy, there are additional properties of the original model that the distilled model should preserve to be considered as an appropriate abstraction. In this work, we explore whether distilled TinyBERT models preserve confidence values of the original BERT models, and investigate how this confidence preservation property could guide tuning hyperparameters of the distillation process.

社交媒体平台通过检测帖子和评论的有害内容来防止恶意活动，为此，它们采用大规模的深度神经网络语言模型进行情感分析和内容理解。为了克服模型复杂、参数众多的缺点，行业专家采用了知识蒸馏压缩技术，其中训练了一个蒸馏模型以复制原始模型的分类行为。本研究探讨蒸馏的TinyBERT模型是否保持了原始BERT模型的置信度值，并探讨这种置信度保持特性如何引导蒸馏过程中的超参数调整。

知识蒸馏抽象中的信心保持性质