Embedding models have been an effective learning paradigm for
high-dimensional data. However, one open issue of embedding models is that
their representations (latent factors) often result in large parameter space.
We observe that existing distributed training frameworks face a scalability
issue of embedding models since updating and retrieving the shared embedding
parameters from servers usually dominates the training cycle. In this paper, we
propose HET, a new system framework that significantly improves the scalability
of huge embedding model training. We embrace skewed popularity distributions of
embeddings as a performance opportunity and leverage it to address the
communication bottleneck with an embedding cache. To ensure consistency across
the caches, we incorporate a new consistency model into HET design, which
provides fine-grained consistency guarantees on a per-embedding basis. Compared
to previous work that only allows staleness for read operations, HET also
utilizes staleness for write operations. Evaluations on six representative
tasks show that HET achieves up to 88% embedding communication reductions and
up to 20.68x performance speedup over the state-of-the-art baselines.

该研究提出了一种名为 HET 的系统框架，旨在解决嵌入式模型大规模训练的可伸缩性问题，通过使用嵌入式缓存和新的一致性模型，可以实现具有细粒度一致性保证的高效通信，相较于现有技术，HET 能够减少高达 88% 的嵌入式通信，并实现高达 20.68 倍的性能加速。

HET: 利用缓存启用的分布式框架扩展大型嵌入式模型训练

HET: Scaling out Huge Embedding Model Training via Cache-enabled  Distributed Framework

We consider the problem of learning a Bayesian network or directed acyclic
graph (DAG) model from observational data. A number of constraint-based,
score-based and hybrid algorithms have been developed for this purpose. For
constraint-based methods, statistical consistency guarantees typically rely on
the faithfulness assumption, which has been show to be restrictive especially
for graphs with cycles in the skeleton. However, there is only limited work on
consistency guarantees for score-based and hybrid algorithms and it has been
unclear whether consistency guarantees can be proven under weaker conditions
than the faithfulness assumption. In this paper, we propose the sparsest
permutation (SP) algorithm. This algorithm is based on finding the causal
ordering of the variables that yields the sparsest DAG. We prove that this new
score-based method is consistent under strictly weaker conditions than the
faithfulness assumption. We also demonstrate through simulations on small DAGs
that the SP algorithm compares favorably to the constraint-based PC and SGS
algorithms as well as the score-based Greedy Equivalence Search and hybrid
Max-Min Hill-Climbing method. In the Gaussian setting, we prove that our
algorithm boils down to finding the permutation of the variables with sparsest
Cholesky decomposition for the inverse covariance matrix. Using this
connection, we show that in the oracle setting, where the true covariance
matrix is known, the SP algorithm is in fact equivalent to $\ell_0$-penalized
maximum likelihood estimation.

该研究提出了一种新的基于分数的算法（SP 算法）来学习贝叶斯网络或有向无环图（DAG）模型，该算法在弱于忠实性假设的条件下具有一致性保证，并且在一些小 DAG 的模拟中表现出与其他算法相比的优越性。