Motivated by the recent empirical success of incorporating public data into
differentially private learning, we theoretically investigate how a shared
representation learned from public data can improve private learning. We
explore two common scenarios of transfer learning for linear regression, both
of which assume the public and private tasks (regression vectors) share a
low-rank subspace in a high-dimensional space. In the first single-task
transfer scenario, the goal is to learn a single model shared across all users,
each corresponding to a row in a dataset. We provide matching upper and lower
bounds showing that our algorithm achieves the optimal excess risk within a
natural class of algorithms that search for the linear model within the given
subspace estimate. In the second scenario of multitask model personalization,
we show that with sufficient public data, users can avoid private coordination,
as purely local learning within the given subspace achieves the same utility.
Taken together, our results help to characterize the benefits of public data
across common regimes of private transfer learning.

通过实证研究公共数据如何融入差分隐私学习并从公共数据中学习共享表示，我们理论上研究了共享表示如何改进私有学习，并探索了线性回归中的两种常见迁移学习场景，结果帮助了解公共数据在不同私有迁移学习中的益处。

利用公共的表示进行私有的迁移学习

Leveraging Public Representations for Private Transfer Learning

Sentences are important semantic units of natural language. A generic,
distributional representation of sentences that can capture the latent
semantics is beneficial to multiple downstream applications. We observe a
simple geometry of sentences -- the word representations of a given sentence
(on average 10.23 words in all SemEval datasets with a standard deviation 4.84)
roughly lie in a low-rank subspace (roughly, rank 4). Motivated by this
observation, we represent a sentence by the low-rank subspace spanned by its
word vectors. Such an unsupervised representation is empirically validated via
semantic textual similarity tasks on 19 different datasets, where it
outperforms the sophisticated neural network models, including skip-thought
vectors, by 15% on average.

研究发现句子的词向量表示一般都集中在一个低秩子空间内，通过基于该子空间的无监督表示法在 19 个数据集上测试，平均优于 skip-thought vectors 等神经网络模型 15%。