Animals are able to rapidly infer from limited experience when sets of state
action pairs have equivalent reward and transition dynamics. On the other hand,
modern reinforcement learning systems must painstakingly learn through trial
and error that sets of state action pairs are value equivalent -- requiring an
often prohibitively large amount of samples from their environment. MDP
homomorphisms have been proposed that reduce the observed MDP of an environment
to an abstract MDP, which can enable more sample efficient policy learning.
Consequently, impressive improvements in sample efficiency have been achieved
when a suitable MDP homomorphism can be constructed a priori -- usually by
exploiting a practioner's knowledge of environment symmetries. We propose a
novel approach to constructing a homomorphism in discrete action spaces, which
uses a partial model of environment dynamics to infer which state action pairs
lead to the same state -- reducing the size of the state-action space by a
factor equal to the cardinality of the action space. We call this method
equivalent effect abstraction. In a gridworld setting, we demonstrate
empirically that equivalent effect abstraction can improve sample efficiency in
a model-free setting and planning efficiency for modelbased approaches.
Furthermore, we show on cartpole that our approach outperforms an existing
method for learning homomorphisms, while using 33x less training data.

提出了一种新方法，即等效效果抽象，该方法利用环境动态的部分模型推断导致相同状态的状态动作对，从而将状态动作空间的大小减少一个等于动作空间基数的因子，以提高采样效率和规划效率。在网格世界环境下，通过实验证明，等效效果抽象可以在模型自由设置和基于模型的方法的规划效率中提高采样效率。此外，通过在车杆环境中进行实验，还表明本方法比现有方法更优秀，在使用 33 倍少的训练数据的情况下实现了更好的表现。