We investigate the behavior of methods that use linear projections to remove
information about a concept from a language representation, and we consider the
question of what happens to a dataset transformed by such a method. A
theoretical analysis and experiments on real-world and synthetic data show that
these methods inject strong statistical dependencies into the transformed
datasets. After applying such a method, the representation space is highly
structured: in the transformed space, an instance tends to be located near
instances of the opposite label. As a consequence, the original labeling can in
some cases be reconstructed by applying an anti-clustering method.

通过线性投影方法探究将一个概念从语言表达中移除后的数据集的行为，表明这些方法会在转换后的数据集中注入强大的统计依赖性，并导致转换空间高度结构化，可能通过反聚类方法重构原始标签。

基于投影的概念去除方法对数据集的影响

What Happens to a Dataset Transformed by a Projection-based Concept  Removal Method?

Cross-lingual word embeddings (CLWE) are often evaluated on bilingual lexicon
induction (BLI). Recent CLWE methods use linear projections, which underfit the
training dictionary, to generalize on BLI. However, underfitting can hinder
generalization to other downstream tasks that rely on words from the training
dictionary. We address this limitation by retrofitting CLWE to the training
dictionary, which pulls training translation pairs closer in the embedding
space and overfits the training dictionary. This simple post-processing step
often improves accuracy on two downstream tasks, despite lowering BLI test
accuracy. We also retrofit to both the training dictionary and a synthetic
dictionary induced from CLWE, which sometimes generalizes even better on
downstream tasks. Our results confirm the importance of fully exploiting
training dictionary in downstream tasks and explains why BLI is a flawed CLWE
evaluation.

该研究利用 CLWE 实现双语词典识别，提出引入修正策略来缓解因线性投影方法欠拟合训练字典而导致下游任务中推广性不足问题，并通过实验证明了该方法对训练字典的重要性及 BLI 评价不足的原因。

为什么过拟合并不总是不好：将跨语言词向量改装成词典

Why Overfitting Isn't Always Bad: Retrofitting Cross-Lingual Word  Embeddings to Dictionaries

X in R^D has mean zero and finite second moments. We show that there is a
precise sense in which almost all linear projections of X into R^d (for d < D)
look like a scale-mixture of spherical Gaussians -- specifically, a mixture of
distributions N(0, sigma^2 I_d) where the weight of the particular sigma
component is P (| X |^2 = sigma^2 D). The extent of this effect depends upon
the ratio of d to D, and upon a particular coefficient of eccentricity of X's
distribution. We explore this result in a variety of experiments.

通过实验探索了线性投影、球形高斯分布、比例混合、偏心系数等相关问题。

投影的集中定理

A concentration theorem for projections

We study the problem of reconstructing a sparse signal from a limited number
of its linear projections when a part of its support is known, although the
known part may contain some errors. The ``known" part of the support, denoted
T, may be available from prior knowledge. Alternatively, in a problem of
recursively reconstructing time sequences of sparse spatial signals, one may
use the support estimate from the previous time instant as the ``known" part.
The idea of our proposed solution (modified-CS) is to solve a convex relaxation
of the following problem: find the signal that satisfies the data constraint
and is sparsest outside of T. We obtain sufficient conditions for exact
reconstruction using modified-CS. These are much weaker than those needed for
compressive sensing (CS) when the sizes of the unknown part of the support and
of errors in the known part are small compared to the support size. An
important extension called Regularized Modified-CS (RegModCS) is developed
which also uses prior signal estimate knowledge. Simulation comparisons for
both sparse and compressible signals are shown.

本研究考虑利用已知部分的支持的信息来重构稀疏信号，提出了一种基于凸松弛的算法来获得除部分支持已知的信号外最稀疏的信号，以及基于先前信号估计知识的重构方法，并进行了模拟比较。