Multi-task learning (MTL) has been widely used in recommender systems,
wherein predicting each type of user feedback on items (e.g, click, purchase)
are treated as individual tasks and jointly trained with a unified model. Our
key observation is that the prediction results of each task may contain
task-specific knowledge about user's fine-grained preference towards items.
While such knowledge could be transferred to benefit other tasks, it is being
overlooked under the current MTL paradigm. This paper, instead, proposes a
Cross-Task Knowledge Distillation framework that attempts to leverage
prediction results of one task as supervised signals to teach another task.
However, integrating MTL and KD in a proper manner is non-trivial due to
several challenges including task conflicts, inconsistent magnitude and
requirement of synchronous optimization. As countermeasures, we 1) introduce
auxiliary tasks with quadruplet loss functions to capture cross-task
fine-grained ranking information and avoid task conflicts, 2) design a
calibrated distillation approach to align and distill knowledge from auxiliary
tasks, and 3) propose a novel error correction mechanism to enable and
facilitate synchronous training of teacher and student models. Comprehensive
experiments are conducted to verify the effectiveness of our framework in
real-world datasets.

提出了一种交叉任务知识蒸馏框架，通过引入辅助任务和校准蒸馏方法，解决了多任务学习中涉及的任务冲突、不一致和同步优化的问题，在实现细粒度推荐、跨任务知识转移等方面有显著效果。

多任务推荐中的跨任务知识蒸馏

Cross-Task Knowledge Distillation in Multi-Task Recommendation

Distributed training of deep learning models on large-scale training data is
typically conducted with asynchronous stochastic optimization to maximize the
rate of updates, at the cost of additional noise introduced from asynchrony. In
contrast, the synchronous approach is often thought to be impractical due to
idle time wasted on waiting for straggling workers. We revisit these
conventional beliefs in this paper, and examine the weaknesses of both
approaches. We demonstrate that a third approach, synchronous optimization with
backup workers, can avoid asynchronous noise while mitigating for the worst
stragglers. Our approach is empirically validated and shown to converge faster
and to better test accuracies.

本文研究了分布式训练深度学习模型时采用同步优化的可行性，发现同步优化虽然会浪费一些等待时间，但采用备用工作者的方式可以避免异步噪声并加速收敛，同时能够在保证准确性的前提下提高测试准确率。