This paper introduces the 2nd place solution for the Riiid! Answer Correctness Prediction in Kaggle, the world's largest data science competition website. This competition was held from October 16, 2020, to January 7, 2021, with 3395 teams and 4387 competitors. The main insights and contributions of this paper are as follows. (i) We pointed out existing Transformer-based models are suffering from a problem that the information which their query/key/value can contain is limited. To solve this problem, we proposed a method that uses LSTM to obtain query/key/value and verified its effectiveness. (ii) We pointed out 'inter-container' leakage problem, which happens in datasets where questions are sometimes served together. To solve this problem, we showed special indexing/masking techniques that are useful when using RNN-variants and Transformer. (iii) We found additional hand-crafted features are effective to overcome the limits of Transformer, which can never consider the samples older than the sequence length.

本文介绍了在 Kaggle 进行的 Riiid! Answer Correctness Prediction 竞赛中排名第二的解决方案，其中指出了现有的基于 Transformer 的模型存在问题，并提出了使用 LSTM 得到查询/键/值的方法。同时，该文还发现针对特定数据集的索引和掩码技术以及额外的手工特征对克服 Transformer 的限制非常有效。

LSTM-SAKT: 基于LSTM编码的类SAKT Transformer用于知识追踪