We study diverse skill discovery in reward-free environments, aiming to
discover all possible skills in simple grid-world environments where prior
methods have struggled to succeed. This problem is formulated as mutual
training of skills using an intrinsic reward and a discriminator trained to
predict a skill given its trajectory. Our initial solution replaces the
standard one-vs-all (softmax) discriminator with a one-vs-one (all pairs)
discriminator and combines it with a novel intrinsic reward function and a
dropout regularization technique. The combined approach is named APART: Diverse
Skill Discovery using All Pairs with Ascending Reward and Dropout. We
demonstrate that APART discovers all the possible skills in grid worlds with
remarkably fewer samples than previous works. Motivated by the empirical
success of APART, we further investigate an even simpler algorithm that
achieves maximum skills by altering VIC, rescaling its intrinsic reward, and
tuning the temperature of its softmax discriminator. We believe our findings
shed light on the crucial factors underlying success of skill discovery
algorithms in reinforcement learning.

在无奖励环境中研究多样化技能发现，使用内在奖励和一个通过轨迹预测技能的判别器进行技能的相互训练，通过使用所有对组合的判别器、新颖的内在奖励函数和辍学正则化技术的方式，取代了标准的一对多（softmax）判别器，所提出的组合方法命名为 APART，在简单的网格环境中比以前的方法大大减少了样本数量，研究探究了更简单的算法，通过改变 VIC、重新调整内在奖励和调整 softmax 判别器的温度来实现最大技能，我们相信我们的研究结果揭示了强化学习中技能发现算法成功的关键因素。

APART: 采用升序奖励和丢弃法的多样化技能发现

APART: Diverse Skill Discovery using All Pairs with Ascending Reward and  DropouT

In many real-world applications, safety constraints for reinforcement
learning (RL) algorithms are either unknown or not explicitly defined. We
propose a framework that concurrently learns safety constraints and optimal RL
policies in such environments, supported by theoretical guarantees. Our
approach merges a logically-constrained RL algorithm with an evolutionary
algorithm to synthesize signal temporal logic (STL) specifications. The
framework is underpinned by theorems that establish the convergence of our
joint learning process and provide error bounds between the discovered policy
and the true optimal policy. We showcased our framework in grid-world
environments, successfully identifying both acceptable safety constraints and
RL policies while demonstrating the effectiveness of our theorems in practice.

提出了一种结合逻辑约束强化学习算法和进化算法的框架，用于在不确定或未明确定义安全约束的环境中并发地学习安全约束和最优 RL 策略，并且该框架以理论保证为支撑，成功地在 grid-world 环境中识别出可接受的安全约束和 RL 策略，以及证明了我们的方法的实践效果。