Recent years have seen significant advances in explainable AI as the need to
understand deep learning models has gained importance with the increased
emphasis on trust and ethics in AI. Comprehensible models for sequential
decision tasks are a particular challenge as they require understanding not
only individual predictions but a series of predictions that interact with
environmental dynamics. We present a framework for learning comprehensible
models of sequential decision tasks in which agent strategies are characterized
using temporal logic formulas. Given a set of agent traces, we first cluster
the traces using a novel embedding method that captures frequent action
patterns. We then search for logical formulas that explain the agent strategies
in the different clusters. We evaluate our framework on combat scenarios in
StarCraft II (SC2), using traces from a handcrafted expert policy and a trained
reinforcement learning agent. We implemented a feature extractor for SC2
environments that extracts traces as sequences of high-level features
describing both the state of the environment and the agent's local behavior
from agent replays. We further designed a visualization tool depicting the
movement of units in the environment that helps understand how different task
conditions lead to distinct agent behavior patterns in each trace cluster.
Experimental results show that our framework is capable of separating agent
traces into distinct groups of behaviors for which our approach to strategy
inference produces consistent, meaningful, and easily understood strategy
descriptions.

本研究提出了一个框架，用于学习顺序决策任务的可理解模型，通过时间逻辑公式表征代理策略，并使用一个嵌入方法对代理足迹进行聚类，得出在不同的聚类中解释代理策略的逻辑公式，通过编写一个特征提取器和一个可视化工具，对在 StarCraft II 中的战斗场景进行了评估，实验结果表明，本框架可以将代理足迹分为不同的行为组，并为每个行为组提供一致、有意义且易于理解的策略描述。