The big data software stack based on Apache Spark and Hadoop has become
mission critical in many enterprises. Performance of Spark and Hadoop jobs
depends on a large number of configuration settings. Manual tuning is expensive
and brittle. There have been prior efforts to develop on-line and off-line
automatic tuning approaches to make the big data stack less dependent on manual
tuning. These, however, demonstrated only modest performance improvements with
very simple, single-user workloads on small data sets. This paper presents
KERMIT - the autonomic architecture for big data capable of automatically
tuning Apache Spark and Hadoop on-line, and achieving performance results 30%
faster than rule-of-thumb tuning by a human administrator and up to 92% as fast
as the fastest possible tuning established by performing an exhaustive search
of the tuning parameter space. KERMIT can detect important workload changes
with up to 99% accuracy, and predict future workload types with up to 96%
accuracy. It is capable of identifying and classifying complex multi-user
workloads without being explicitly trained on examples of these workloads. It
does not rely on the past workload history to predict the future workload
classes and their associated performance. KERMIT can identify and learn new
workload classes, and adapt to workload drift, without human intervention.

KERMIT 是一种自主架构，可以自动调整 Apache Spark 和 Hadoop，并能识别和分类复杂的多用户工作负载，以达到比人工管理员的启发式调优快 30％，并且比通过对调优参数空间进行透彻搜索获得的最快调优结果快高达 92％。