Despite the popularity of Model Compression and Multitask Learning, how to
effectively compress a multitask model has been less thoroughly analyzed due to
the challenging entanglement of tasks in the parameter space. In this paper, we
propose DiSparse, a simple, effective, and first-of-its-kind multitask pruning
and sparse training scheme. We consider each task independently by
disentangling the importance measurement and take the unanimous decisions among
all tasks when performing parameter pruning and selection. Our experimental
results demonstrate superior performance on various configurations and settings
compared to popular sparse training and pruning methods. Besides the
effectiveness in compression, DiSparse also provides a powerful tool to the
multitask learning community. Surprisingly, we even observed better performance
than some dedicated multitask learning methods in several cases despite the
high model sparsity enforced by DiSparse. We analyzed the pruning masks
generated with DiSparse and observed strikingly similar sparse network
architecture identified by each task even before the training starts. We also
observe the existence of a "watershed" layer where the task relatedness sharply
drops, implying no benefits in continued parameters sharing. Our code and
models will be available at:
this https URL.

本文提出 DiSparse 算法，一个多任务模型简化与稀疏训练方法，可分解任务，采用一致的决策进行参数修剪和选择，并在多种配置下，表现出优于常见稀疏训练和修剪方法的性能。