Current semantic segmentation methods focus only on mining "local" context,
i.e., dependencies between pixels within individual images, by
context-aggregation modules (e.g., dilated convolution, neural attention) or
structure-aware optimization criteria (e.g., IoU-like loss). However, they
ignore "global" context of the training data, i.e., rich semantic relations
between pixels across different images. Inspired by the recent advance in
unsupervised contrastive representation learning, we propose a pixel-wise
contrastive framework for semantic segmentation in the fully supervised
setting. The core idea is to enforce pixel embeddings belonging to a same
semantic class to be more similar than embeddings from different classes. It
raises a pixel-wise metric learning paradigm for semantic segmentation, by
explicitly exploring the structures of labeled pixels, which were rarely
explored before. Our method can be effortlessly incorporated into existing
segmentation frameworks without extra overhead during testing. We
experimentally show that, with famous segmentation models (i.e., DeepLabV3,
HRNet, OCR) and backbones (i.e., ResNet, HR-Net), our method brings consistent
performance improvements across diverse datasets (i.e., Cityscapes,
PASCAL-Context, COCO-Stuff, CamVid). We expect this work will encourage our
community to rethink the current de facto training paradigm in fully supervised
semantic segmentation.

提出了一个利用对比学习实现像素级别度量学习的语义分割框架，能够从全局上探索标记像素的结构，并在多个数据集上带来了显著的性能改进。

探索跨图像像素对比用于语义分割

Exploring Cross-Image Pixel Contrast for Semantic Segmentation

Almost all existing deep learning approaches for semantic segmentation tackle
this task as a pixel-wise classification problem. Yet humans understand a scene
not in terms of pixels, but by decomposing it into perceptual groups and
structures that are the basic building blocks of recognition. This motivates us
to propose an end-to-end pixel-wise metric learning approach that mimics this
process. In our approach, the optimal visual representation determines the
right segmentation within individual images and associates segments with the
same semantic classes across images. The core visual learning problem is
therefore to maximize the similarity within segments and minimize the
similarity between segments. Given a model trained this way, inference is
performed consistently by extracting pixel-wise embeddings and clustering, with
the semantic label determined by the majority vote of its nearest neighbors
from an annotated set.
As a result, we present the SegSort, as a first attempt using deep learning
for unsupervised semantic segmentation, achieving $76\%$ performance of its
supervised counterpart. When supervision is available, SegSort shows consistent
improvements over conventional approaches based on pixel-wise softmax training.
Additionally, our approach produces more precise boundaries and consistent
region predictions. The proposed SegSort further produces an interpretable
result, as each choice of label can be easily understood from the retrieved
nearest segments.

本文提出了基于深度学习的端到端的像素度量学习方法，通过优化相似性函数来实现图像内分割和跨图像语义分组，并且采用像素嵌入和聚类的方式进行一致的推理和预测，该方法在无监督的语义分割方面取得了很好的性能并且具有更精确的边界和一致的区域预测，并且可以产生可解释的结果。