Vision-language contrastive learning frameworks like CLIP enable learning
representations from natural language supervision, and provide strong zero-shot
classification capabilities. However, due to the nature of the supervisory
signal in these paradigms, they lack the ability to learn localized features,
leading to degraded performance on dense prediction tasks like segmentation and
detection. On the other hand, self-supervised learning methods have shown the
ability to learn granular representations, complementing the high-level
features in vision-language training. In this work, we present Harmony, a
framework that combines vision-language training with discriminative and
generative self-supervision to learn visual features that can be generalized
across vision downstream tasks. Our framework is specifically designed to work
on web-scraped data by not relying on negative examples and addressing the
one-to-one correspondence issue using soft CLIP targets generated by an EMA
model. We comprehensively evaluate Harmony across various vision downstream
tasks and find that it significantly outperforms the baseline CLIP and the
previously leading joint self and weakly-supervised methods, MaskCLIP and SLIP.
Specifically, when comparing against these methods, Harmony shows superior
performance in fine-tuning and zero-shot classification on ImageNet-1k,
semantic segmentation on ADE20K, and both object detection and instance
segmentation on MS-COCO, when pre-training a ViT-S/16 on CC3M. We also show
that Harmony outperforms other self-supervised learning methods like iBOT and
MAE across all tasks evaluated. On this https URL our
code is publicly available.

本研究中，我们提出了一种名为 Harmony 的框架，它结合了视觉 - 语言训练和辨别性自监督以及生成性自监督，以学习能够在视觉下游任务中被泛化的视觉特征。我们全面评估了 Harmony 在各种视觉下游任务上的性能，并发现它在 ImageNet-1k 上的微调和零样本分类、ADE20K 上的语义分割以及 MS-COCO 上的目标检测和实例分割等任务中明显优于基线 CLIP 和先前领先的联合自监督和弱监督方法 MaskCLIP 和 SLIP。与其他自监督学习方法（如 iBOT 和 MAE）相比，我们还展示了 Harmony 在所有评估任务上的优越性能。

Harmony: 一种联合自监督和弱监督框架，用于学习通用的视觉表示

Harmony: A Joint Self-Supervised and Weakly-Supervised Framework for  Learning General Purpose Visual Representations

Organizations with a large user base, such as Samsung and Google, can
potentially benefit from collecting and mining users' data. However, doing so
raises privacy concerns, and risks accidental privacy breaches with serious
consequences. Local differential privacy (LDP) techniques address this problem
by only collecting randomized answers from each user, with guarantees of
plausible deniability; meanwhile, the aggregator can still build accurate
models and predictors by analyzing large amounts of such randomized data. So
far, existing LDP solutions either have severely restricted functionality, or
focus mainly on theoretical aspects such as asymptotical bounds rather than
practical usability and performance. Motivated by this, we propose Harmony, a
practical, accurate and efficient system for collecting and analyzing data from
smart device users, while satisfying LDP. Harmony applies to multi-dimensional
data containing both numerical and categorical attributes, and supports both
basic statistics (e.g., mean and frequency estimates), and complex machine
learning tasks (e.g., linear regression, logistic regression and SVM
classification). Experiments using real data confirm Harmony's effectiveness.

Harmony 是一种实用、准确和高效的系统，支持从智能设备用户收集和分析多维度数据以及应用于基本的统计学和复杂的机器学习任务，并满足本地差分隐私技术的要求，以解决数据收集可能引起的隐私问题。

使用本地差分隐私对智能设备用户数据进行收集和分析

Collecting and Analyzing Data from Smart Device Users with Local  Differential Privacy

Our goal is to be able to build a generative model from a deep neural network
architecture to try to create music that has both harmony and melody and is
passable as music composed by humans. Previous work in music generation has
mainly been focused on creating a single melody. More recent work on polyphonic
music modeling, centered around time series probability density estimation, has
met some partial success. In particular, there has been a lot of work based off
of Recurrent Neural Networks combined with Restricted Boltzmann Machines
(RNN-RBM) and other similar recurrent energy based models. Our approach,
however, is to perform end-to-end learning and generation with deep neural nets
alone.

本研究尝试使用深度神经网络建立生成模型，以生成既有和声和旋律，并且足以通过人类作曲的音乐，并利用端到端学习和生成的方法。