Recent studies suggest that the existing neural models have difficulty
handling repeated items in sequential recommendation tasks. However, our
understanding of this difficulty is still limited. In this study, we
substantially advance this field by identifying a major source of the problem:
the single hidden state embedding and static item embeddings in the output
softmax layer. Specifically, the similarity structure of the global item
embeddings in the softmax layer sometimes forces the single hidden state
embedding to be close to new items when copying is a better choice, while
sometimes forcing the hidden state to be close to the items from the input
inappropriately. To alleviate the problem, we adapt the recently-proposed
softmax alternatives such as softmax-CPR to sequential recommendation tasks and
demonstrate that the new softmax architectures unleash the capability of the
neural encoder on learning when to copy and when to exclude the items from the
input sequence. By only making some simple modifications on the output softmax
layer for SASRec and GRU4Rec, softmax-CPR achieves consistent improvement in 12
datasets. With almost the same model size, our best method not only improves
the average NDCG@10 of GRU4Rec in 5 datasets with duplicated items by 10%
(4%-17% individually) but also improves 7 datasets without duplicated items by
24% (8%-39%)!

最近的研究表明，现有的神经模型在处理顺序推荐任务中的重复项目时存在困难。然而，我们对这个困难的理解仍然有限。通过识别问题的主要来源 —— 输出 softmax 层中的单一隐藏状态嵌入和静态项目嵌入的相似性结构，本研究在该领域取得了显著的进展。具体地说，当复制是更好的选择时，全局项目嵌入在 softmax 层中的相似性结构有时会导致单一隐藏状态嵌入接近新项目，而有时会不适当地导致隐藏状态接近输入中的项目。为了缓解这个问题，我们将最近提出的 softmax 替代方法（如 softmax-CPR）应用于顺序推荐任务，并证明了新的 softmax 架构释放了神经编码器在学习何时复制和何时排除输入序列中的项目方面的能力。通过仅对 SASRec 和 GRU4Rec 的输出 softmax 层进行一些简单的修改，softmax-CPR 在 12 个数据集中实现了持续的改进。与几乎相同的模型大小相比，我们的最佳方法不仅将 GRU4Rec 在有重复项目的 5 个数据集中的平均 NDCG@10 提高了 10%（个别情况下提高了 4%-17%），还将没有重复项目的 7 个数据集的结果提高了 24%（提高了 8%-39%）！

拷贝还是不拷贝：神经序列推荐模型中的输出 Softmax 层是一个关键问题

To Copy, or not to Copy; That is a Critical Issue of the Output Softmax  Layer in Neural Sequential Recommenders

Large language models (LLMs) have recently received significant attention for
their exceptional capabilities. Despite extensive efforts in developing
general-purpose LLMs that can be utilized in various natural language
processing (NLP) tasks, there has been less research exploring their potential
in recommender systems. In this paper, we propose a novel framework, named
PALR, which aiming to combine user history behaviors (such as clicks,
purchases, ratings, etc.) with LLMs to generate user preferred items.
Specifically, we first use user/item interactions as guidance for candidate
retrieval. Then we adopt a LLM-based ranking model to generate recommended
items. Unlike existing approaches that typically adopt general-purpose LLMs for
zero/few-shot recommendation testing or training on small-sized language models
(with less than 1 billion parameters), which cannot fully elicit LLMs'
reasoning abilities and leverage rich item side parametric knowledge, we
fine-tune a 7 billion parameters LLM for the ranking purpose. This model takes
retrieval candidates in natural language format as input, with instruction
which explicitly asking to select results from input candidates during
inference. Our experimental results demonstrate that our solution outperforms
state-of-the-art models on various sequential recommendation tasks.

本文提出了一种名为 PALR 的新框架，旨在将用户历史行为与大型语言模型相结合，生成用户喜欢的商品。实验结果表明，与现有方法相比，我们的解决方案在各种连续推荐任务上表现优异。