Distributed learning on the edge often comprises self-centered devices (SCD)
which learn local tasks independently and are unwilling to contribute to the
performance of other SDCs. How do we achieve forward transfer at zero cost for
the single SCDs? We formalize this problem as a Distributed Continual Learning
scenario, where SCD adapt to local tasks and a CL model consolidates the
knowledge from the resulting stream of models without looking at the SCD's
private data. Unfortunately, current CL methods are not directly applicable to
this scenario. We propose Data-Agnostic Consolidation (DAC), a novel double
knowledge distillation method that consolidates the stream of SC models without
using the original data. DAC performs distillation in the latent space via a
novel Projected Latent Distillation loss. Experimental results show that DAC
enables forward transfer between SCDs and reaches state-of-the-art accuracy on
Split CIFAR100, CORe50 and Split TinyImageNet, both in reharsal-free and
distributed CL scenarios. Somewhat surprisingly, even a single
out-of-distribution image is sufficient as the only source of data during
consolidation.

本文提出了一种 Data-Agnostic Consolidation (DAC) 双知识蒸馏方法，通过新的投影潜在蒸馏损失函数在潜在空间中蒸馏模型，从而在单个设备上实现了零成本的前向传递，进而在分布式持续学习情境中实现流模型的知识整合和转移，最终在各类数据集中达到了最先进的效果。