The sheer number of nodes continues to increase in todays supercomputers, the
first half of Trinity alone contains more than 9400 compute nodes. Since the
speed of todays clusters are limited by the slowest nodes, it more important
than ever to identify slow nodes, improve their performance if it can be done,
and assure minimal usage of slower nodes during performance critical runs. This
is an ongoing maintenance task that occurs on a regular basis and, therefore,
it is important to minimize the impact upon its users by assessing and
addressing slow performing nodes and mitigating their consequences while
minimizing down time. These issues can be solved, in large part, through a
systematic application of fast running hardware assessment tests, the
application of Machine Learning, and making use of performance data to increase
efficiency of large clusters. Proxy applications utilizing both MPI and OpenMP
were developed to produce data as a substitute for long runtime applications to
evaluate node performance. Machine learning is applied to identify
underperforming nodes, and policies are being discussed to both minimize the
impact of underperforming nodes and increase the efficiency of the system. In
this paper, I will describe the process used to produce quickly performing
proxy tests, consider various methods to isolate the outliers, and produce
ordered lists for use in scheduling to accomplish this task.

使用快速运行的代理测试来评估超级计算机中的性能问题，运用机器学习识别性能差的节点，并讨论减少其影响和提高系统效率的策略。

利用机器学习、代理应用和调度优先级在 Trinity 上优化性能

Optimizing Performance on Trinity Utilizing Machine Learning, Proxy  Applications and Scheduling Priorities

As Exascale computing becomes a reality, the energy needs of compute nodes in
cloud data centers will continue to grow. A common approach to reducing this
energy demand is to limit the power consumption of hardware components when
workloads are experiencing bottlenecks elsewhere in the system. However,
designing a resource controller capable of detecting and limiting power
consumption on-the-fly is a complex issue and can also adversely impact
application performance. In this paper, we explore the use of Reinforcement
Learning (RL) to design a power capping policy on cloud compute nodes using
observations on current power consumption and instantaneous application
performance (heartbeats). By leveraging the Argo Node Resource Management (NRM)
software stack in conjunction with the Intel Running Average Power Limit (RAPL)
hardware control mechanism, we design an agent to control the maximum supplied
power to processors without compromising on application performance. Employing
a Proximal Policy Optimization (PPO) agent to learn an optimal policy on a
mathematical model of the compute nodes, we demonstrate and evaluate using the
STREAM benchmark how a trained agent running on actual hardware can take
actions by balancing power consumption and application performance.

当 Exascale 计算成为现实时，云数据中心的计算节点能源需求将继续增长。在系统其他方面经历瓶颈时，降低硬件组件的功耗是减少能源需求的常见方法之一。然而，设计一个能够实时检测和限制功耗的资源控制器是一个复杂的问题，可能会对应用性能产生不利影响。本文中，我们探索使用强化学习（RL）设计云计算节点上的功耗限制策略，利用当前功耗和瞬时应用性能（心跳）的观测。通过将 Argo Node Resource Management（NRM）软件堆栈与 Intel Running Average Power Limit（RAPL）硬件控制机制相结合，设计一个代理程序来控制处理器的最大供电功率，而不会影响应用性能。利用 Proximal Policy Optimization（PPO）代理程序在计算节点的数学模型上学习最优策略，我们使用 STREAM 基准测试演示和评估了在实际硬件上运行的经过训练的代理程序如何通过平衡功耗和应用性能来采取行动。

数据中心计算节点功耗降低的性能感知增强学习方法

A Reinforcement Learning Approach for Performance-aware Reduction in  Power Consumption of Data Center Compute Nodes

With the rise of AI in recent years and the increase in complexity of the
models, the growing demand in computational resources is starting to pose a
significant challenge. The need for higher compute power is being met with
increasingly more potent accelerators and the use of large compute clusters.
However, the gain in prediction accuracy from large models trained on
distributed and accelerated systems comes at the price of a substantial
increase in energy demand, and researchers have started questioning the
environmental friendliness of such AI methods at scale. Consequently, energy
efficiency plays an important role for AI model developers and infrastructure
operators alike. The energy consumption of AI workloads depends on the model
implementation and the utilized hardware. Therefore, accurate measurements of
the power draw of AI workflows on different types of compute nodes is key to
algorithmic improvements and the design of future compute clusters and
hardware. To this end, we present measurements of the energy consumption of two
typical applications of deep learning models on different types of compute
nodes. Our results indicate that 1. deriving energy consumption directly from
runtime is not accurate, but the consumption of the compute node needs to be
considered regarding its composition; 2. neglecting accelerator hardware on
mixed nodes results in overproportional inefficiency regarding energy
consumption; 3. energy consumption of model training and inference should be
considered separately - while training on GPUs outperforms all other node types
regarding both runtime and energy consumption, inference on CPU nodes can be
comparably efficient. One advantage of our approach is that the information on
energy consumption is available to all users of the supercomputer, enabling an
easy transfer to other workloads alongside a raise in user-awareness of energy
consumption.

近年来人工智能的兴起和模型复杂度的增加已开始带来计算资源需求上的挑战，本文针对大规模 AI 方法的能源效率问题，通过对不同计算节点上深度学习模型进行测量，说明了在算法层面和未来计算集群和硬件设计方面，准确测量 AI 工作量的功耗需求之重要性，以及模型训练和推理的能耗差异，提供了降低成本、提高用户意识的解决方案。