Identifying the most suitable variables to represent the state is a
fundamental challenge in Reinforcement Learning (RL). These variables must
efficiently capture the information necessary for making optimal decisions. In
order to address this problem, in this paper, we introduce the Transfer Entropy
Redundancy Criterion (TERC), an information-theoretic criterion, which
determines if there is \textit{entropy transferred} from state variables to
actions during training. We define an algorithm based on TERC that provably
excludes variables from the state that have no effect on the final performance
of the agent, resulting in more sample efficient learning. Experimental results
show that this speed-up is present across three different algorithm classes
(represented by tabular Q-learning, Actor-Critic, and Proximal Policy
Optimization (PPO)) in a variety of environments. Furthermore, to highlight the
differences between the proposed methodology and the current state-of-the-art
feature selection approaches, we present a series of controlled experiments on
synthetic data, before generalizing to real-world decision-making tasks. We
also introduce a representation of the problem that compactly captures the
transfer of information from state variables to actions as Bayesian networks.

在本论文中，我们介绍了一种用于强化学习（RL）的信息论准则 Transfer Entropy Redundancy Criterion (TERC)，该准则确定了训练过程中是否从状态变量到动作变量传递了信息熵。基于 TERC 的算法能够排除对智能体最终性能没有影响的状态变量，从而提高样本利用效率。实验证明这种加速方式在不同算法类别和各种环境中都存在。为了突出与现有特征选择方法的区别，我们在合成数据上进行了一系列对照实验，并推广到现实世界的决策任务。我们还引入了一种用贝叶斯网络紧凑地捕捉状态变量到动作变量的信息传递的问题表征。