In the reinforcement learning literature, there are many algorithms developed
for either Contextual Bandit (CB) or Markov Decision Processes (MDP)
environments. However, when deploying reinforcement learning algorithms in the
real world, even with domain expertise, it is often difficult to know whether
it is appropriate to treat a sequential decision making problem as a CB or an
MDP. In other words, do actions affect future states, or only the immediate
rewards? Making the wrong assumption regarding the nature of the environment
can lead to inefficient learning, or even prevent the algorithm from ever
learning an optimal policy, even with infinite data. In this work we develop an
online algorithm that uses a Bayesian hypothesis testing approach to learn the
nature of the environment. Our algorithm allows practitioners to incorporate
prior knowledge about whether the environment is that of a CB or an MDP, and
effectively interpolate between classical CB and MDP-based algorithms to
mitigate against the effects of misspecifying the environment. We perform
simulations and demonstrate that in CB settings our algorithm achieves lower
regret than MDP-based algorithms, while in non-bandit MDP settings our
algorithm is able to learn the optimal policy, often achieving comparable
regret to MDP-based algorithms.

研究了强化学习中决策问题的环境类型不确定性问题，提出了一种基于贝叶斯假设检验方法的在线算法，可以在上下文探索和马尔科夫过程决策算法之间互相转换来适应环境类型，以避免不合适的环境假设引起的低效问题。

使用贝叶斯方法学习马尔可夫决策过程中的 Bandit 结构

A Bayesian Approach to Learning Bandit Structure in Markov Decision Processes

Usually one compares the accuracy of two competing classifiers via null
hypothesis significance tests (nhst). Yet the nhst tests suffer from important
shortcomings, which can be overcome by switching to Bayesian hypothesis
testing. We propose a Bayesian hierarchical model which jointly analyzes the
cross-validation results obtained by two classifiers on multiple data sets. It
returns the posterior probability of the accuracies of the two classifiers
being practically equivalent or significantly different. A further strength of
the hierarchical model is that, by jointly analyzing the results obtained on
all data sets, it reduces the estimation error compared to the usual approach
of averaging the cross-validation results obtained on a given data set.

本研究提出了一种贝叶斯分层模型，通过对多个数据集上两个分类器的交叉验证结果进行联合分析，从而返回两个分类器准确度在实际上是否等价或显著不同的后验概率，并减少了估计误差。

贝叶斯分层建模统计比较分类器

Statistical comparison of classifiers through Bayesian hierarchical  modelling

A/B testing is one of the most successful applications of statistical theory
in modern Internet age. One problem of Null Hypothesis Statistical Testing
(NHST), the backbone of A/B testing methodology, is that experimenters are not
allowed to continuously monitor the result and make decision in real time. Many
people see this restriction as a setback against the trend in the technology
toward real time data analytics. Recently, Bayesian Hypothesis Testing, which
intuitively is more suitable for real time decision making, attracted growing
interest as an alternative to NHST. While corrections of NHST for the
continuous monitoring setting are well established in the existing literature
and known in A/B testing community, the debate over the issue of whether
continuous monitoring is a proper practice in Bayesian testing exists among
both academic researchers and general practitioners. In this paper, we formally
prove the validity of Bayesian testing with continuous monitoring when proper
stopping rules are used, and illustrate the theoretical results with concrete
simulation illustrations. We point out common bad practices where stopping
rules are not proper and also compare our methodology to NHST corrections.
General guidelines for researchers and practitioners are also provided.

本文证明了在使用适当停止规则的情况下，贝叶斯检验与连续监控的有效性，并通过具体模拟实例说明了理论结果。文章指出了常见的不良实践，比较了该方法与 NHST 校正，并提供了研究人员和实践者的一般指导方针。