Portfolio management is a fundamental problem in finance. It involves
periodic reallocations of assets to maximize the expected returns within an
appropriate level of risk exposure. Deep reinforcement learning (RL) has been
considered a promising approach to solving this problem owing to its strong
capability in sequential decision making. However, due to the non-stationary
nature of financial markets, applying RL techniques to portfolio optimization
remains a challenging problem. Extracting trading knowledge from various expert
strategies could be helpful for agents to accommodate the changing markets. In
this paper, we propose MetaTrader, a novel two-stage RL-based approach for
portfolio management, which learns to integrate diverse trading policies to
adapt to various market conditions. In the first stage, MetaTrader incorporates
an imitation learning objective into the reinforcement learning framework.
Through imitating different expert demonstrations, MetaTrader acquires a set of
trading policies with great diversity. In the second stage, MetaTrader learns a
meta-policy to recognize the market conditions and decide on the most proper
learned policy to follow. We evaluate the proposed approach on three real-world
index datasets and compare it to state-of-the-art baselines. The empirical
results demonstrate that MetaTrader significantly outperforms those baselines
in balancing profits and risks. Furthermore, thorough ablation studies validate
the effectiveness of the components in the proposed approach.

本文提出了一种基于元学习和深度增强学习的投资组合管理方法 MetaTrader，通过模仿不同的专家策略来获取适应不同市场的交易策略，并学习元策略来决定最适合的策略。实证结果表明，MetaTrader 在平衡利润和风险方面显著优于现有的基线模型，并通过彻底的消融研究验证了该方法组成部分的有效性。