Although Reinforcement Learning (RL) algorithms acquire sequential behavioral
patterns through interactions with the environment, their effectiveness in
noisy and high-dimensional scenarios typically relies on specific structural
priors. In this paper, we propose a novel and general Structural Information
principles-based framework for effective Decision-Making, namely SIDM,
approached from an information-theoretic perspective. This paper presents a
specific unsupervised partitioning method that forms vertex communities in the
state and action spaces based on their feature similarities. An aggregation
function, which utilizes structural entropy as the vertex weight, is devised
within each community to obtain its embedding, thereby facilitating
hierarchical state and action abstractions. By extracting abstract elements
from historical trajectories, a directed, weighted, homogeneous transition
graph is constructed. The minimization of this graph's high-dimensional entropy
leads to the generation of an optimal encoding tree. An innovative two-layer
skill-based learning mechanism is introduced to compute the common path entropy
of each state transition as its identified probability, thereby obviating the
requirement for expert knowledge. Moreover, SIDM can be flexibly incorporated
into various single-agent and multi-agent RL algorithms, enhancing their
performance. Finally, extensive evaluations on challenging benchmarks
demonstrate that, compared with SOTA baselines, our framework significantly and
consistently improves the policy's quality, stability, and efficiency up to
32.70%, 88.26%, and 64.86%, respectively.

该研究提出了一种基于结构信息原则的有效决策框架，通过信息论的视角，利用一种特定的无监督划分方法生成状态和动作的抽象表示，构建高效的转换图和编码树，引入了基于技能的学习机制，从而改善了单一智能体和多智能体强化学习算法的性能。

基于结构信息原则的有效强化学习

Effective Reinforcement Learning Based on Structural Information  Principles

Demonstration-guided reinforcement learning (RL) is a promising approach for
learning complex behaviors by leveraging both reward feedback and a set of
target task demonstrations. Prior approaches for demonstration-guided RL treat
every new task as an independent learning problem and attempt to follow the
provided demonstrations step-by-step, akin to a human trying to imitate a
completely unseen behavior by following the demonstrator's exact muscle
movements. Naturally, such learning will be slow, but often new behaviors are
not completely unseen: they share subtasks with behaviors we have previously
learned. In this work, we aim to exploit this shared subtask structure to
increase the efficiency of demonstration-guided RL. We first learn a set of
reusable skills from large offline datasets of prior experience collected
across many tasks. We then propose Skill-based Learning with Demonstrations
(SkiLD), an algorithm for demonstration-guided RL that efficiently leverages
the provided demonstrations by following the demonstrated skills instead of the
primitive actions, resulting in substantial performance improvements over prior
demonstration-guided RL approaches. We validate the effectiveness of our
approach on long-horizon maze navigation and complex robot manipulation tasks.

本文提出了一种基于示教引导的强化学习方法，该方法通过提取任务中共享的子任务结构，从而显著提高学习效率，并且在迷宫导航和复杂机器人操纵任务上进行了验证。