We present a new class of structured reinforcement learning policy-architectures, Implicit Two-Tower (ITT) policies, where the actions are chosen based on the attention scores of their learnable latent representations with those of the input states. By explicitly disentangling action from state processing in the policy stack, we achieve two main goals: substantial computational gains and better performance. Our architectures are compatible with both: discrete and continuous action spaces. By conducting tests on 15 environments from OpenAI Gym and DeepMind Control Suite, we show that ITT-architectures are particularly suited for blackbox/evolutionary optimization and the corresponding policy training algorithms outperform their vanilla unstructured implicit counterparts as well as commonly used explicit policies. We complement our analysis by showing how techniques such as hashing and lazy tower updates, critically relying on the two-tower structure of ITTs, can be applied to obtain additional computational improvements.

该论文提出了一种新的结构强化学习策略架构，Implicit Two-Tower（ITT）策略，并在OpenAI Gym和DeepMind Control Suite的15个环境上进行了测试，表明ITT-architectures特别适用于黑盒/进化优化，并且相应的策略训练算法优于其未经结构化改进的隐式对照组以及常用的显式策略。

隐式双塔策略