Offline reinforcement-learning (RL) algorithms learn to make decisions using
a given, fixed training dataset without online data collection. This problem
setting is captivating because it holds the promise of utilizing previously
collected datasets without any costly or risky interaction with the
environment. However, this promise also bears the drawback of this setting as
the restricted dataset induces uncertainty because the agent can encounter
unfamiliar sequences of states and actions that the training data did not
cover. To mitigate the destructive uncertainty effects, we need to balance the
aspiration to take reward-maximizing actions with the incurred risk due to
incorrect ones. In financial economics, modern portfolio theory (MPT) is a
method that risk-averse investors can use to construct diversified portfolios
that maximize their returns without unacceptable levels of risk. We propose
integrating MPT into the agent's decision-making process, presenting a new
simple-yet-highly-effective risk-aware planning algorithm for offline RL. Our
algorithm allows us to systematically account for the \emph{estimated quality}
of specific actions and their \emph{estimated risk} due to the uncertainty. We
show that our approach can be coupled with the Transformer architecture to
yield a state-of-the-art planner, which maximizes the return for offline RL
tasks. Moreover, our algorithm reduces the variance of the results
significantly compared to conventional Transformer decoding, which results in a
much more stable algorithm -- a property that is essential for the offline RL
setting, where real-world exploration and failures can be costly or dangerous.

本研究提出了一种将现代投资组合理论（MPT）与强化学习相结合、针对离线数据训练进行风险管理的算法，通过集成 MPT, 该算法能够更好地解决离线强化学习中存在的不确定性问题，并在使用 Transformer 结构的情况下达到最优解。

华尔街树搜索：离线强化学习的风险感知规划

Wall Street Tree Search: Risk-Aware Planning for Offline Reinforcement Learning

Asset allocation is an investment strategy that aims to balance risk and
reward by constantly redistributing the portfolio's assets according to certain
goals, risk tolerance, and investment horizon. Unfortunately, there is no
simple formula that can find the right allocation for every individual. As a
result, investors may use different asset allocations' strategy to try to
fulfil their financial objectives. In this work, we conduct an extensive
benchmark study to determine the efficacy and reliability of a number of
optimization techniques. In particular, we focus on traditional approaches
based on Modern Portfolio Theory, and on machine-learning approaches based on
deep reinforcement learning. We assess the model's performance under different
market tendency, i.e., both bullish and bearish markets. For reproducibility,
we provide the code implementation code in this repository.

本研究探讨了在不同市场趋势下，基于现代投资组合理论和深度强化学习的优化技术在资产配置中的效果和可靠性，并提供了代码实现。

资产配置：从马尔科维茨到深度强化学习

Asset Allocation: From Markowitz to Deep Reinforcement Learning

We investigate a hybrid quantum-classical solution method to the
mean-variance portfolio optimization problems. Starting from real financial
data statistics and following the principles of the Modern Portfolio Theory, we
generate parametrized samples of portfolio optimization problems that can be
related to quadratic binary optimization forms programmable in the analog
D-Wave Quantum Annealer 2000Q. The instances are also solvable by an
industry-established Genetic Algorithm approach, which we use as a classical
benchmark. We investigate several options to run the quantum computation
optimally, ultimately discovering that the best results in terms of expected
time-to-solution as a function of number of variables for the hardest instances
set are obtained by seeding the quantum annealer with a solution candidate
found by a greedy local search and then performing a reverse annealing
protocol. The optimized reverse annealing protocol is found to be more than 100
times faster than the corresponding forward quantum annealing on average.

本研究基于现代投资组合理论原则，利用实际的金融数据统计生成参数化的投资组合优化问题样本，将问题转换成可由模拟模拟模拟模拟模拟的二次二进制优化问题，研究了多种运行量子计算的方式，最终发现采用 “贪心局部搜索 + 反向退火” 方案的时间效率最优，比正向量子退火平均快 100 倍以上。