This paper proposes a new sequential model learning architecture to solve
partially observable Markov decision problems. Rather than compressing
sequential information at every timestep as in conventional recurrent neural
network-based methods, the proposed architecture generates a latent variable in
each data block with a length of multiple timesteps and passes the most
relevant information to the next block for policy optimization. The proposed
blockwise sequential model is implemented based on self-attention, making the
model capable of detailed sequential learning in partial observable settings.
The proposed model builds an additional learning network to efficiently
implement gradient estimation by using self-normalized importance sampling,
which does not require the complex blockwise input data reconstruction in the
model learning. Numerical results show that the proposed method significantly
outperforms previous methods in various partially observable environments.

本文提出了一种新的序列模型学习体系结构，旨在解决部分可观测的马可夫决策问题。该模型依靠自注意力实现分块序列模型，能够在部分可观测的环境下进行详细的序列学习，而不需要对模型学习中的输入数据进行复杂的块重构。同时，该模型通过使用自正规化重要抽样构建附加的学习网络，并在各种部分可观测的环境中显著优于先前的方法。