To make reinforcement learning more sample efficient, we need better credit
assignment methods that measure an action's influence on future rewards.
Building upon Hindsight Credit Assignment (HCA), we introduce Counterfactual
Contribution Analysis (COCOA), a new family of model-based credit assignment
algorithms. Our algorithms achieve precise credit assignment by measuring the
contribution of actions upon obtaining subsequent rewards, by quantifying a
counterfactual query: "Would the agent still have reached this reward if it had
taken another action?". We show that measuring contributions w.r.t. rewarding
states, as is done in HCA, results in spurious estimates of contributions,
causing HCA to degrade towards the high-variance REINFORCE estimator in many
relevant environments. Instead, we measure contributions w.r.t. rewards or
learned representations of the rewarding objects, resulting in gradient
estimates with lower variance. We run experiments on a suite of problems
specifically designed to evaluate long-term credit assignment capabilities. By
using dynamic programming, we measure ground-truth policy gradients and show
that the improved performance of our new model-based credit assignment methods
is due to lower bias and variance compared to HCA and common baselines. Our
results demonstrate how modeling action contributions towards rewarding
outcomes can be leveraged for credit assignment, opening a new path towards
sample-efficient reinforcement learning.

本文介绍了一种基于 Counterfactual Contribution Analysis（COCOA）的信用分配算法，在衡量行动对未来奖励的影响时，通过量化一个反事实的查询来实现精确的信用分配，即 “如果智能体采取另一个行动，它仍然能达到这个奖励吗？”，在评估长期信用分配能力的一套问题上进行了实验，并通过与 HCA 和常见基线比较表明，模型信用分配算法的改进性能由于较低的偏差和方差，为实现样本高效强化学习开辟了新的道路。

透过反事实贡献分析的长期信用分配问题

Would I have gotten that reward? Long-term credit assignment by  counterfactual contribution analysis

There are two distinct approaches to solving reinforcement learning problems,
namely, searching in value function space and searching in policy space.
Temporal difference methods and evolutionary algorithms are well-known examples
of these approaches. Kaelbling, Littman and Moore recently provided an
informative survey of temporal difference methods. This article focuses on the
application of evolutionary algorithms to the reinforcement learning problem,
emphasizing alternative policy representations, credit assignment methods, and
problem-specific genetic operators. Strengths and weaknesses of the
evolutionary approach to reinforcement learning are presented, along with a
survey of representative applications.

本文针对强化学习问题，重点介绍了进化算法在该问题上的应用，以及替代策略表述、信誉分配方法和问题特定遗传算子等方面的思考，在总结代表性应用的同时呈现了进化方法在强化学习中的优势和劣势。