This paper addresses the challenge of a particular class of noisy state
observations in Markov Decision Processes (MDPs), a common issue in various
real-world applications. We focus on modeling this uncertainty through a
confusion matrix that captures the probabilities of misidentifying the true
state. Our primary goal is to estimate the inherent measurement noise, and to
this end, we propose two novel algorithmic approaches. The first, the method of
second-order repetitive actions, is designed for efficient noise estimation
within a finite time window, providing identifiable conditions for system
analysis. The second approach comprises a family of Bayesian algorithms, which
we thoroughly analyze and compare in terms of performance and limitations. We
substantiate our theoretical findings with simulations, demonstrating the
effectiveness of our methods in different scenarios, particularly highlighting
their behavior in environments with varying stationary distributions. Our work
advances the understanding of reinforcement learning in noisy environments,
offering robust techniques for more accurate state estimation in MDPs.

通过建模不确定性来解决马尔可夫决策过程（MDPs）中一类特定的嘈杂状态观测问题，提出了两种新的算法方法，一种是用于有限时间窗口内有效噪声估计的二阶重复动作方法，另一种是一组贝叶斯算法，通过模拟实验证实了这些方法在不同场景下尤其在具有不同稳定分布的环境中的有效性。