Selecting the right resources for big data analytics jobs is hard because of
the wide variety of configuration options like machine type and cluster size.
As poor choices can have a significant impact on resource efficiency, cost, and
energy usage, automated approaches are gaining popularity. Most existing
methods rely on profiling recurring workloads to find near-optimal solutions
over time. Due to the cold-start problem, this often leads to lengthy and
costly profiling phases. However, big data analytics jobs across users can
share many common properties: they often operate on similar infrastructure,
using similar algorithms implemented in similar frameworks. The potential in
sharing aggregated profiling runs to collaboratively address the cold start
problem is largely unexplored.
We present Karasu, an approach to more efficient resource configuration
profiling that promotes data sharing among users working with similar
infrastructures, frameworks, algorithms, or datasets. Karasu trains lightweight
performance models using aggregated runtime information of collaborators and
combines them into an ensemble method to exploit inherent knowledge of the
configuration search space. Moreover, Karasu allows the optimization of
multiple objectives simultaneously. Our evaluation is based on performance data
from diverse workload executions in a public cloud environment. We show that
Karasu is able to significantly boost existing methods in terms of performance,
search time, and cost, even when few comparable profiling runs are available
that share only partial common characteristics with the target job.

Karasu 是一种更高效的资源配置剖析方法，通过促进类似基础设施、框架、算法或数据集的用户之间的数据共享，训练轻量级性能模型并将它们组合成一个集成方法，以开发配置搜索空间中固有的知识，同时优化多个目标，从而显著提升现有方法在性能、搜索时间和成本方面的表现。

Karasu：大数据分析高效集群配置的合作方法

Karasu: A Collaborative Approach to Efficient Cluster Configuration for  Big Data Analytics

The cloud is still a popular platform for distributed deep learning (DL)
training jobs since resource sharing in the cloud can improve resource
utilization and reduce overall costs. However, such sharing also brings
multiple challenges for DL training jobs, e.g., high-priority jobs could
impact, even interrupt, low-priority jobs. Meanwhile, most existing distributed
DL training systems require users to configure the resources (i.e., the number
of nodes and resources like CPU and memory allocated to each node) of jobs
manually before job submission and can not adjust the job's resources during
the runtime. The resource configuration of a job deeply affect this job's
performance (e.g., training throughput, resource utilization, and completion
rate). However, this usually leads to poor performance of jobs since users fail
to provide optimal resource configuration in most cases. \system~is a
distributed DL framework can auto-configure a DL job's initial resources and
dynamically tune the job's resources to win the better performance. With
elastic capability, \system~can effectively adjusts the resources of a job when
there are performance issues detected or a job fails because of faults or
eviction. Evaluations results show \system~can outperform manual well-tuned
resource configurations. Furthermore, in the production Kubernetes cluster of
\company, \system~reduces the medium of job completion time by 31\%, and
improves the job completion rate by 6\%, CPU utilization by 15\%, and memory
utilization by 20\% compared with manual configuration.

系统是一个自动配置深度学习作业资源的分布式深度学习框架，具有动态调整作业资源以获得更好性能的弹性能力。