Function-as-a-Service is a cloud computing paradigm offering an event-driven
execution model to applications. It features serverless attributes by
eliminating resource management responsibilities from developers and offers
transparent and on-demand scalability of applications. Typical serverless
applications have stringent response time and scalability requirements and
therefore rely on deployed services to provide quick and fault-tolerant
feedback to clients. However, the FaaS paradigm suffers from cold starts as
there is a non-negligible delay associated with on-demand function
initialization. This work focuses on reducing the frequency of cold starts on
the platform by using Reinforcement Learning. Our approach uses Q-learning and
considers metrics such as function CPU utilization, existing function
instances, and response failure rate to proactively initialize functions in
advance based on the expected demand. The proposed solution was implemented on
Kubeless and was evaluated using a normalised real-world function demand trace
with matrix multiplication as the workload. The results demonstrate a
favourable performance of the RL-based agent when compared to Kubeless' default
policy and function keep-alive policy by improving throughput by up to 8.81%
and reducing computation load and resource wastage by up to 55% and 37%,
respectively, which is a direct outcome of reduced cold starts.

通过使用强化学习减少冷启动频率，以提高函数即时初始化效果的解决方案的实现与评估。

强化学习（RL）增强冷启动频率减少在无服务器计算中的应用

Reinforcement Learning (RL) Augmented Cold Start Frequency Reduction in  Serverless Computing

Function-as-a-Service (FaaS) introduces a lightweight, function-based cloud
execution model that finds its relevance in applications like IoT-edge data
processing and anomaly detection. While CSP offer a near-infinite function
elasticity, these applications often experience fluctuating workloads and
stricter performance constraints. A typical CSP strategy is to empirically
determine and adjust desired function instances, "autoscaling", based on
monitoring-based thresholds such as CPU or memory, to cope with demand and
performance. However, threshold configuration either requires expert knowledge,
historical data or a complete view of environment, making autoscaling a
performance bottleneck lacking an adaptable solution.RL algorithms are proven
to be beneficial in analysing complex cloud environments and result in an
adaptable policy that maximizes the expected objectives. Most realistic cloud
environments usually involve operational interference and have limited
visibility, making them partially observable. A general solution to tackle
observability in highly dynamic settings is to integrate Recurrent units with
model-free RL algorithms and model a decision process as a POMDP. Therefore, in
this paper, we investigate a model-free Recurrent RL agent for function
autoscaling and compare it against the model-free Proximal Policy Optimisation
(PPO) algorithm. We explore the integration of a LSTM network with the
state-of-the-art PPO algorithm to find that under our experimental and
evaluation settings, recurrent policies were able to capture the environment
parameters and show promising results for function autoscaling. We further
compare a PPO-based autoscaling agent with commercially used threshold-based
function autoscaling and posit that a LSTM-based autoscaling agent is able to
improve throughput by 18%, function execution by 13% and account for 8.4% more
function instances.

通过将模型无关的循环强化学习（Recurrent RL）代理与最先进的 PPO 算法相结合，我们研究了用于函数自动缩放的模型无关的 Recurrent RL 代理，并将其与基于阈值的函数自动缩放进行了比较，发现循环策略能够捕捉环境参数并在函数自动缩放方面显示出有希望的结果。除此之外，我们将基于 PPO 的自动缩放代理与商业使用的基于阈值的函数自动缩放进行了比较，并认为基于 LSTM 的自动缩放代理能够提高吞吐量 18％，函数执行速度 13％，并支持多出 8.4％的函数实例。