Planning-based reinforcement learning has shown strong performance in tasks
in discrete and low-dimensional continuous action spaces. However, planning
usually brings significant computational overhead for decision-making, and
scaling such methods to high-dimensional action spaces remains challenging. To
advance efficient planning for high-dimensional continuous control, we propose
Trajectory Autoencoding Planner (TAP), which learns low-dimensional latent
action codes with a state-conditional VQ-VAE. The decoder of the VQ-VAE thus
serves as a novel dynamics model that takes latent actions and current state as
input and reconstructs long-horizon trajectories. During inference time, given
a starting state, TAP searches over discrete latent actions to find
trajectories that have both high probability under the training distribution
and high predicted cumulative reward. Empirical evaluation in the offline RL
setting demonstrates low decision latency which is indifferent to the growing
raw action dimensionality. For Adroit robotic hand manipulation tasks with
high-dimensional continuous action space, TAP surpasses existing model-based
methods by a large margin and also beats strong model-free actor-critic
baselines.

该论文提出了一种基于 Trajectory Autoencoding Planner (TAP) 的规划算法，通过使用低维潜在动作编码和状态条件 VQ-VAE 模型，以及搜索离散的潜在动作来发现高累积奖励可能性的轨迹，并在高维持续行为空间中超越了现有的模型和策略的基线结果。