Learning sentence embeddings is a fundamental problem in natural language
processing. While existing research primarily focuses on enhancing the quality
of sentence embeddings, the exploration of sentence embedding dimensions is
limited. Here we present a comprehensive and empirical analysis of the
dimensionality of sentence embeddings. First, we demonstrate that the optimal
dimension of sentence embeddings is usually smaller than the default value.
Subsequently, to compress the dimension of sentence embeddings with minimum
performance degradation, we identify two components contributing to the overall
performance loss: the encoder's performance loss and the pooler's performance
loss. Therefore, we propose a two-step training method for sentence
representation learning models, wherein the encoder and the pooler are
optimized separately to mitigate the overall performance loss in low-dimension
scenarios. Experimental results on seven STS tasks and seven sentence
classification tasks demonstrate that our method significantly improves the
performance of low-dimensional sentence embeddings.

通过维度分析，我们提出了一种双步训练方法，优化编码器和池化器以减轻低维场景中的整体性能损失，从而显著提高低维句子嵌入的性能。

关于句子嵌入的维度

On the Dimensionality of Sentence Embeddings

In this paper, we initiate a rigorous study of the phenomenon of
low-dimensional adversarial perturbations (LDAPs) in classification. Unlike the
classical setting, these perturbations are limited to a subspace of dimension
$k$ which is much smaller than the dimension $d$ of the feature space. The case
$k=1$ corresponds to so-called universal adversarial perturbations (UAPs;
Moosavi-Dezfooli et al., 2017). First, we consider binary classifiers under
generic regularity conditions (including ReLU networks) and compute analytical
lower-bounds for the fooling rate of any subspace. These bounds explicitly
highlight the dependence of the fooling rate on the pointwise margin of the
model (i.e., the ratio of the output to its $L_2$ norm of its gradient at a
test point), and on the alignment of the given subspace with the gradients of
the model w.r.t. inputs. Our results provide a rigorous explanation for the
recent success of heuristic methods for efficiently generating low-dimensional
adversarial perturbations. Finally, we show that if a decision-region is
compact, then it admits a universal adversarial perturbation with $L_2$ norm
which is $\sqrt{d}$ times smaller than the typical $L_2$ norm of a data point.
Our theoretical results are confirmed by experiments on both synthetic and real
data.

本文研究了低维敌对干扰、通用性敌对干扰等与分类有关主题，并给出了关于低维敌对干扰欺骗率的严格分析及与最近的启发式方法的成功之间的解释。实验证明了理论上的结果。

低维对抗扰动的起源

Origins of Low-dimensional Adversarial Perturbations

Knowledge Graph Embedding (KGE) is a popular method for KG reasoning and
training KGEs with higher dimension are usually preferred since they have
better reasoning capability. However, high-dimensional KGEs pose huge
challenges to storage and computing resources and are not suitable for
resource-limited or time-constrained applications, for which faster and cheaper
reasoning is necessary. To address this problem, we propose DualDE, a knowledge
distillation method to build low-dimensional student KGE from pre-trained
high-dimensional teacher KGE. DualDE considers the dual-influence between the
teacher and the student. In DualDE, we propose a soft label evaluation
mechanism to adaptively assign different soft label and hard label weights to
different triples, and a two-stage distillation approach to improve the
student's acceptance of the teacher. Our DualDE is general enough to be applied
to various KGEs. Experimental results show that our method can successfully
reduce the embedding parameters of a high-dimensional KGE by 7 times - 15 times
and increase the inference speed by 2 times - 6 times while retaining a high
performance. We also experimentally prove the effectiveness of our soft label
evaluation mechanism and two-stage distillation approach via ablation study.

提出一种双重教导方法 DualDE，用于提高知识图谱嵌入的推理效率，通过软标签评估机制和两阶段蒸馏方法，可以将高维 KGE 减少 7-15 倍，推理速度提高 2-6 倍。

DualDE: 双重蒸馏知识图谱嵌入以加速和降低推理成本

DualDE: Dually Distilling Knowledge Graph Embedding for Faster and  Cheaper Reasoning

In the quest for efficient and robust reinforcement learning methods, both
model-free and model-based approaches offer advantages. In this paper we
propose a new way of explicitly bridging both approaches via a shared
low-dimensional learned encoding of the environment, meant to capture
summarizing abstractions. We show that the modularity brought by this approach
leads to good generalization while being computationally efficient, with
planning happening in a smaller latent state space. In addition, this approach
recovers a sufficient low-dimensional representation of the environment, which
opens up new strategies for interpretable AI, exploration and transfer
learning.

本文提出一种新的方法，通过共享基于低维学习的环境编码来明确地连接无模型和有模型的强化学习方法，该方法能够捕捉到总结性抽象，同时具有模块化的特点，因此具有良好的泛化能力和计算效率，并在较小的潜在状态空间中进行计划。此外，此方法还能恢复足够低维的环境表示，从而为可解释的人工智能、探索和迁移学习开辟了新的策略。