Several distributed frameworks have been developed to scale Graph Neural
Networks (GNNs) on billion-size graphs. On several benchmarks, we observe that
the graph partitions generated by these frameworks have heterogeneous data
distributions and class imbalance, affecting convergence, and resulting in
lower performance than centralized implementations. We holistically address
these challenges and develop techniques that reduce training time and improve
accuracy. We develop an Edge-Weighted partitioning technique to improve the
micro average F1 score (accuracy) by minimizing the total entropy. Furthermore,
we add an asynchronous personalization phase that adapts each compute-host's
model to its local data distribution. We design a class-balanced sampler that
considerably speeds up convergence. We implemented our algorithms on the
DistDGL framework and observed that our training techniques scale much better
than the existing training approach. We achieved a (2-3x) speedup in training
time and 4\% improvement on average in micro-F1 scores on 5 large graph
benchmarks compared to the standard baselines.

通过改进分布式框架和训练技术，提高了大规模图神经网络的训练效率和准确性，减少了数据分布和类别不均衡引起的性能下降，并在基准测试中实现了训练时间加速和 F1 分数的提升。

熵感知训练方法用于快速准确的分布式图神经网络

Entropy Aware Training for Fast and Accurate Distributed GNN

Task-based distributed frameworks (e.g., Ray, Dask, Hydro) have become
increasingly popular for distributed applications that contain asynchronous and
dynamic workloads, including asynchronous gradient descent, reinforcement
learning, and model serving. As more data-intensive applications move to run on
top of task-based systems, collective communication efficiency has become an
important problem. Unfortunately, traditional collective communication
libraries (e.g., MPI, Horovod, NCCL) are an ill fit, because they require the
communication schedule to be known before runtime and they do not provide fault
tolerance.
We design and implement Hoplite, an efficient and fault-tolerant collective
communication layer for task-based distributed systems. Our key technique is to
compute data transfer schedules on the fly and execute the schedules
efficiently through fine-grained pipelining. At the same time, when a task
fails, the data transfer schedule adapts quickly to allow other tasks to keep
making progress. We apply Hoplite to a popular task-based distributed
framework, Ray. We show that Hoplite speeds up asynchronous stochastic gradient
descent, reinforcement learning, and serving an ensemble of machine learning
models that are difficult to execute efficiently with traditional collective
communication by up to 7.8x, 3.9x, and 3.3x, respectively.

Hoplite 是一种高效的、容错的、用于面向任务的分布式系统的集体通信层，它能够实时计算数据传输计划并通过细粒度流水线方式执行以处理异步和动态工作负载，同时提供容错能力，可应用于异步随机梯度下降等应用，提高传统集体通信效率达 7.8 倍。

Hoplite：面向任务型分布式系统的高效容错集体通信

Hoplite: Efficient and Fault-Tolerant Collective Communication for  Task-Based Distributed Systems

Distributed frameworks are gaining increasingly widespread use in
applications that process large amounts of data. One important example
application is large scale similarity search, for which Locality Sensitive
Hashing (LSH) has emerged as the method of choice, specially when the data is
high-dimensional. At its core, LSH is based on hashing the data points to a
number of buckets such that similar points are more likely to map to the same
buckets. To guarantee high search quality, the LSH scheme needs a rather large
number of hash tables. This entails a large space requirement, and in the
distributed setting, with each query requiring a network call per hash bucket
look up, this also entails a big network load. The Entropy LSH scheme proposed
by Panigrahy significantly reduces the number of required hash tables by
looking up a number of query offsets in addition to the query itself. While
this improves the LSH space requirement, it does not help with (and in fact
worsens) the search network efficiency, as now each query offset requires a
network call. In this paper, focusing on the Euclidian space under $l_2$ norm
and building up on Entropy LSH, we propose the distributed Layered LSH scheme,
and prove that it exponentially decreases the network cost, while maintaining a
good load balance between different machines. Our experiments also verify that
our scheme results in a significant network traffic reduction that brings about
large runtime improvement in real world applications.

本文提出了基于熵局部敏感哈希的分布式分层哈希方案，能够显著减少网络流量，提高实际应用程序的运行效率。