Models applied on real time response task, like click-through rate (CTR)
prediction model, require high accuracy and rigorous response time. Therefore,
top-performing deep models of high depth and complexity are not well suited for
these applications with the limitations on the inference time. In order to
further improve the neural networks' performance given the time and
computational limitations, we propose an approach that exploits a cumbersome
net to help train the lightweight net for prediction. We dub the whole process
rocket launching, where the cumbersome booster net is used to guide the
learning of the target light net throughout the whole training process. We
analyze different loss functions aiming at pushing the light net to behave
similarly to the booster net, and adopt the loss with best performance in our
experiments. We use one technique called gradient block to improve the
performance of the light net and booster net further. Experiments on benchmark
datasets and real-life industrial advertisement data present that our light
model can get performance only previously achievable with more complex models.

本文提出了一种名为 “火箭发射” 的方法，利用笨重网络来指导轻量级网络的学习，并分析了不同的损失函数，采用了最佳性能的损失函数来推动轻量级网络表现得像笨重网络。实验结果表明，这种方法能够让轻量级模型获得之前只能由更复杂的模型才能实现的性能。