In this paper, we consider the supervised pretrained transformer for a class
of sequential decision-making problems. The class of considered problems is a
subset of the general formulation of reinforcement learning in that there is no
transition probability matrix, and the class of problems covers bandits,
dynamic pricing, and newsvendor problems as special cases. Such a structure
enables the use of optimal actions/decisions in the pretraining phase, and the
usage also provides new insights for the training and generalization of the
pretrained transformer. We first note that the training of the transformer
model can be viewed as a performative prediction problem, and the existing
methods and theories largely ignore or cannot resolve the arisen
out-of-distribution issue. We propose a natural solution that includes the
transformer-generated action sequences in the training procedure, and it enjoys
better properties both numerically and theoretically. The availability of the
optimal actions in the considered tasks also allows us to analyze the
properties of the pretrained transformer as an algorithm and explains why it
may lack exploration and how this can be automatically resolved. Numerically,
we categorize the advantages of the pretrained transformer over the structured
algorithms such as UCB and Thompson sampling into three cases: (i) it better
utilizes the prior knowledge in the pretraining data; (ii) it can elegantly
handle the misspecification issue suffered by the structured algorithms; (iii)
for short time horizon such as $T\le50$, it behaves more greedy and enjoys much
better regret than the structured algorithms which are designed for asymptotic
optimality.

本文考虑了一类顺序决策问题的受监督预训练变压器模型，并且提出了一种解决预训练变压器训练及泛化问题的自然方法，该方法通过在训练过程中包含变压器生成的动作序列来提供更好的性能。同时，文章还分析了预训练变压器作为一种算法的特性，解释了其缺乏探索性和如何自动解决此问题，并在数值上将其与结构化算法相比较。