With the rise of large language models (LLMs), recent works have leveraged
LLMs to improve the performance of click-through rate (CTR) prediction.
However, we argue that a critical obstacle remains in deploying LLMs for
practical use: the efficiency of LLMs when processing long textual user
behaviors. As user sequences grow longer, the current efficiency of LLMs is
inadequate for training on billions of users and items. To break through the
efficiency barrier of LLMs, we propose Behavior Aggregated Hierarchical
Encoding (BAHE) to enhance the efficiency of LLM-based CTR modeling.
Specifically, BAHE proposes a novel hierarchical architecture that decouples
the encoding of user behaviors from inter-behavior interactions. Firstly, to
prevent computational redundancy from repeated encoding of identical user
behaviors, BAHE employs the LLM's pre-trained shallow layers to extract
embeddings of the most granular, atomic user behaviors from extensive user
sequences and stores them in the offline database. Subsequently, the deeper,
trainable layers of the LLM facilitate intricate inter-behavior interactions,
thereby generating comprehensive user embeddings. This separation allows the
learning of high-level user representations to be independent of low-level
behavior encoding, significantly reducing computational complexity. Finally,
these refined user embeddings, in conjunction with correspondingly processed
item embeddings, are incorporated into the CTR model to compute the CTR scores.
Extensive experimental results show that BAHE reduces training time and memory
by five times for CTR models using LLMs, especially with longer user sequences.
BAHE has been deployed in a real-world system, allowing for daily updates of 50
million CTR data on 8 A100 GPUs, making LLMs practical for industrial CTR
prediction.

通过提出行为聚合分层编码（BAHE）来提高基于大型语言模型（LLMs）的点击率（CTR）建模的效率，通过利用 LLMs 预训练的浅层获取用户行为的嵌入并与深层进行交互以计算 CTR 得分，从而减少计算复杂性并使 LLMs 实用于工业 CTR 预测。