Recent developments in Generative AI, Computer Vision, and Natural Language
Processing have led to an increased integration of AI models into various
products. This widespread adoption of AI requires significant efforts in
deploying these models in production environments. When hosting machine
learning models for real-time predictions, it is important to meet defined
Service Level Objectives (SLOs), ensuring reliability, minimal downtime, and
optimizing operational costs of the underlying infrastructure. Large machine
learning models often demand GPU resources for efficient inference to meet
SLOs. In the context of these trends, there is growing interest in hosting AI
models in a serverless architecture while still providing GPU access for
inference tasks. This survey aims to summarize and categorize the emerging
challenges and optimization opportunities for large-scale deep learning serving
systems. By providing a novel taxonomy and summarizing recent trends, we hope
that this survey could shed light on new optimization perspectives and motivate
novel works in large-scale deep learning serving systems.

最近在生成人工智能、计算机视觉和自然语言处理方面的进展，已经使得人工智能模型得到了广泛应用。这种广泛应用需要在生产环境中部署这些模型，确保可靠性、最小的停机时间以及优化基础设施的运营成本。大型机器学习模型经常需要 GPU 资源以进行高效推理，以满足服务级别目标。在这些趋势的背景下，人们越来越关注在无服务器架构中托管 AI 模型，并提供推理任务的 GPU 访问。这项调查旨在总结和分类大规模深度学习服务系统中的新兴挑战和优化机会。通过提供新颖的分类法和总结最近的趋势，我们希望这项调查能为大规模深度学习服务系统的新的优化视角提供启示，并激发出创新作品。