Autoscaling is a vital mechanism in cloud computing that supports the
autonomous adjustment of computing resources under dynamic workloads. A primary
goal of autoscaling is to stabilize resource utilization at a desirable level,
thus reconciling the need for resource-saving with the satisfaction of Service
Level Objectives (SLOs). Existing proactive autoscaling methods anticipate the
future workload and scale the resources in advance, whereas the reliability may
suffer from prediction deviations arising from the frequent fluctuations and
noise of cloud workloads; reactive methods rely on real-time system feedback,
while the hysteretic nature of reactive methods could cause violations of the
rigorous SLOs. To this end, this paper presents OptScaler, a hybrid autoscaling
framework that integrates the power of both proactive and reactive methods for
regulating CPU utilization. Specifically, the proactive module of OptScaler
consists of a sophisticated workload prediction model and an optimization
model, where the former provides reliable inputs to the latter for making
optimal scaling decisions. The reactive module provides a self-tuning estimator
of CPU utilization to the optimization model. We embed Model Predictive Control
(MPC) mechanism and robust optimization techniques into the optimization model
to further enhance its reliability. Numerical results have demonstrated the
superiority of both the workload prediction model and the hybrid framework of
OptScaler in the scenario of online services compared to prevalent reactive,
proactive, or hybrid autoscalers. OptScaler has been successfully deployed at
Alipay, supporting the autoscaling of applets in the world-leading payment
platform.

OptScaler 是一个混合自动扩缩容框架，将预测性和反应性方法相结合，通过工作负载预测和资源优化来稳定 CPU 利用率，实现在线服务的可靠自动扩缩容。

OptScaler: 云中稳健自适应扩展的混合主动 - 被动框架

OptScaler: A Hybrid Proactive-Reactive Framework for Robust Autoscaling  in the Cloud

Function-as-a-Service (FaaS) introduces a lightweight, function-based cloud
execution model that finds its relevance in applications like IoT-edge data
processing and anomaly detection. While CSP offer a near-infinite function
elasticity, these applications often experience fluctuating workloads and
stricter performance constraints. A typical CSP strategy is to empirically
determine and adjust desired function instances, "autoscaling", based on
monitoring-based thresholds such as CPU or memory, to cope with demand and
performance. However, threshold configuration either requires expert knowledge,
historical data or a complete view of environment, making autoscaling a
performance bottleneck lacking an adaptable solution.RL algorithms are proven
to be beneficial in analysing complex cloud environments and result in an
adaptable policy that maximizes the expected objectives. Most realistic cloud
environments usually involve operational interference and have limited
visibility, making them partially observable. A general solution to tackle
observability in highly dynamic settings is to integrate Recurrent units with
model-free RL algorithms and model a decision process as a POMDP. Therefore, in
this paper, we investigate a model-free Recurrent RL agent for function
autoscaling and compare it against the model-free Proximal Policy Optimisation
(PPO) algorithm. We explore the integration of a LSTM network with the
state-of-the-art PPO algorithm to find that under our experimental and
evaluation settings, recurrent policies were able to capture the environment
parameters and show promising results for function autoscaling. We further
compare a PPO-based autoscaling agent with commercially used threshold-based
function autoscaling and posit that a LSTM-based autoscaling agent is able to
improve throughput by 18%, function execution by 13% and account for 8.4% more
function instances.

通过将模型无关的循环强化学习（Recurrent RL）代理与最先进的 PPO 算法相结合，我们研究了用于函数自动缩放的模型无关的 Recurrent RL 代理，并将其与基于阈值的函数自动缩放进行了比较，发现循环策略能够捕捉环境参数并在函数自动缩放方面显示出有希望的结果。除此之外，我们将基于 PPO 的自动缩放代理与商业使用的基于阈值的函数自动缩放进行了比较，并认为基于 LSTM 的自动缩放代理能够提高吞吐量 18％，函数执行速度 13％，并支持多出 8.4％的函数实例。

一种智能无服务器函数的深度递归强化学习方法

A Deep Recurrent-Reinforcement Learning Method for Intelligent  AutoScaling of Serverless Functions

Organisations are increasingly putting machine learning models into
production at scale. The increasing popularity of serverless scale-to-zero
paradigms presents an opportunity for deploying machine learning models to help
mitigate infrastructure costs when many models may not be in continuous use. We
will discuss the KFServing project which builds on the KNative serverless
paradigm to provide a serverless machine learning inference solution that
allows a consistent and simple interface for data scientists to deploy their
models. We will show how it solves the challenges of autoscaling GPU based
inference and discuss some of the lessons learnt from using it in production.

本文介绍了利用 KFServing 这一构建在 KNative 无服务器框架之上的服务器化机器学习推理解决方案，在保持简洁一致的界面的同时缓解了基础设施成本与 GPU 推理的自动扩容挑战。