Many transfer problems require re-using previously optimal decisions for
solving new tasks, which suggests the need for learning algorithms that can
modify the mechanisms for choosing certain actions independently of those for
choosing others. However, there is currently no formalism nor theory for how to
achieve this kind of modular credit assignment. To answer this question, we
define modular credit assignment as a constraint on minimizing the algorithmic
mutual information among feedback signals for different decisions. We introduce
what we call the modularity criterion for testing whether a learning algorithm
satisfies this constraint by performing causal analysis on the algorithm
itself. We generalize the recently proposed societal decision-making framework
as a more granular formalism than the Markov decision process to prove that for
decision sequences that do not contain cycles, certain single-step temporal
difference action-value methods meet this criterion while all policy-gradient
methods do not. Empirical evidence suggests that such action-value methods are
more sample efficient than policy-gradient methods on transfer problems that
require only sparse changes to a sequence of previously optimal decisions.

本研究提出了一种模块化信用分配的形式化理论，通过对算法自身的因果分析，将社会决策制定框架作为马尔可夫决策过程的更精细正式框架，证明了一些单步时序差分行为价值方法满足这一约束条件，而所有政策梯度方法都不满足该约束条件，传输学习的实证研究表明这种行为价值方法在要求对先前最优决策序列进行稀疏更改的情况下，比政策梯度方法具有更高的样本效率。

强化学习的模块化：通过算法独立性进行信用分配

Modularity in Reinforcement Learning via Algorithmic Independence in  Credit Assignment

Inferring the causal structure that links n observables is usually based upon
detecting statistical dependences and choosing simple graphs that make the
joint measure Markovian. Here we argue why causal inference is also possible
when only single observations are present.
We develop a theory how to generate causal graphs explaining similarities
between single objects. To this end, we replace the notion of conditional
stochastic independence in the causal Markov condition with the vanishing of
conditional algorithmic mutual information and describe the corresponding
causal inference rules.
We explain why a consistent reformulation of causal inference in terms of
algorithmic complexity implies a new inference principle that takes into
account also the complexity of conditional probability densities, making it
possible to select among Markov equivalent causal graphs. This insight provides
a theoretical foundation of a heuristic principle proposed in earlier work.
We also discuss how to replace Kolmogorov complexity with decidable
complexity criteria. This can be seen as an algorithmic analog of replacing the
empirically undecidable question of statistical independence with practical
independence tests that are based on implicit or explicit assumptions on the
underlying distribution.

本文介绍了一种基于单个观测数据进行因果推断的方法，使用条件算法互信息代替因果马尔科夫条件中的条件随机独立性，解释了单个对象之间相似性的因果图生成，并使用可判定复杂度标准替代 Kolmogorov 复杂度，提出了一种新的因果推断原则，从而可以在 Markov 等价因果图之间进行选择。