The advent of large language models (LLMs) has transformed text-based services, enabling capabilities ranging from real-time translation to AI-driven chatbots. However, existing serving systems primarily focus on optimizing server-side aggregate metrics like token generation throughput, ignoring individual user experience with streamed text. As a result, under high and/or bursty load, a significant number of users can receive unfavorable service quality or poor Quality-of-Experience (QoE). In this paper, we first formally define QoE of text streaming services, where text is delivered incrementally and interactively to users, by considering the end-to-end token delivery process throughout the entire interaction with the user. Thereafter, we propose Andes, a QoE-aware serving system that enhances user experience for LLM-enabled text streaming services. At its core, Andes strategically allocates contended GPU resources among multiple requests over time to optimize their QoE. Our evaluations demonstrate that, compared to the state-of-the-art LLM serving systems like vLLM, Andes improves the average QoE by up to 3.2$\times$ under high request rate, or alternatively, it attains up to 1.6$\times$ higher request rate while preserving high QoE.

Andes是一种以QoE为导向的服务系统，旨在提高基于大型语言模型的文本流式服务的用户体验，并通过在GPU资源之间的战略分配来优化QoE。与vLLM等现有的LLM服务系统相比，Andes在高请求率下可将平均QoE提高高达3.2倍，或者以高QoE为前提达到高达1.6倍的请求率。

Andes: 在基于LLM的文本流媒体服务中定义和增强用户体验质量