Albeit with varying degrees of progress in the field of Semi-Supervised
Semantic Segmentation, most of its recent successes are involved in unwieldy
models and the lightweight solution is still not yet explored. We find that
existing knowledge distillation techniques pay more attention to pixel-level
concepts from labeled data, which fails to take more informative cues within
unlabeled data into account. Consequently, we offer the first attempt to
provide lightweight SSSS models via a novel multi-granularity distillation
(MGD) scheme, where multi-granularity is captured from three aspects: i)
complementary teacher structure; ii) labeled-unlabeled data cooperative
distillation; iii) hierarchical and multi-levels loss setting. Specifically,
MGD is formulated as a labeled-unlabeled data cooperative distillation scheme,
which helps to take full advantage of diverse data characteristics that are
essential in the semi-supervised setting. Image-level semantic-sensitive loss,
region-level content-aware loss, and pixel-level consistency loss are set up to
enrich hierarchical distillation abstraction via structurally complementary
teachers. Experimental results on PASCAL VOC2012 and Cityscapes reveal that MGD
can outperform the competitive approaches by a large margin under diverse
partition protocols. For example, the performance of ResNet-18 and MobileNet-v2
backbone is boosted by 11.5% and 4.6% respectively under 1/16 partition
protocol on Cityscapes. Although the FLOPs of the model backbone is compressed
by 3.4-5.3x (ResNet-18) and 38.7-59.6x (MobileNetv2), the model manages to
achieve satisfactory segmentation results.

提出一种基于多层级知识蒸馏技术的轻量级半监督语义分割模型，采用标注数据和未标注数据的协同蒸馏方案，结合像素级一致性约束、语义感知层次损失和内容感知区域损失等多层次损失策略，从多个角度对模型知识进行精炼，能够在消耗更少的运算资源和更小的模型体积下取得同类模型不可比的性能提升。

面向轻量级半监督语义分割的多粒度蒸馏方案

Multi-Granularity Distillation Scheme Towards Lightweight  Semi-Supervised Semantic Segmentation

Generative Adversarial Networks (GANs) have witnessed prevailing success in
yielding outstanding images, however, they are burdensome to deploy on
resource-constrained devices due to ponderous computational costs and hulking
memory usage. Although recent efforts on compressing GANs have acquired
remarkable results, they still exist potential model redundancies and can be
further compressed. To solve this issue, we propose a novel online
multi-granularity distillation (OMGD) scheme to obtain lightweight GANs, which
contributes to generating high-fidelity images with low computational demands.
We offer the first attempt to popularize single-stage online distillation for
GAN-oriented compression, where the progressively promoted teacher generator
helps to refine the discriminator-free based student generator. Complementary
teacher generators and network layers provide comprehensive and
multi-granularity concepts to enhance visual fidelity from diverse dimensions.
Experimental results on four benchmark datasets demonstrate that OMGD successes
to compress 40x MACs and 82.5X parameters on Pix2Pix and CycleGAN, without loss
of image quality. It reveals that OMGD provides a feasible solution for the
deployment of real-time image translation on resource-constrained devices. Our
code and models are made public at: this https URL

本文提出了一种新颖的在线多粒度蒸馏（OMGD）方案，该方案可用于生成低计算需求的轻量级 GAN，从而实现在资源受限设备上进行实时图像转换，并显示 OMGD 在四个基准数据集上的实验结果。