Reinforcement learning (RL) faces substantial challenges when applied to
real-life problems, primarily stemming from the scarcity of available data due
to limited interactions with the environment. This limitation is exacerbated by
the fact that RL often demands a considerable volume of data for effective
learning. The complexity escalates further when implementing RL in recurrent
spiking networks, where inherent noise introduced by spikes adds a layer of
difficulty. Life-long learning machines must inherently resolve the
plasticity-stability paradox. Striking a balance between acquiring new
knowledge and maintaining stability is crucial for artificial agents. In this
context, we take inspiration from machine learning technology and introduce a
biologically plausible implementation of proximal policy optimization, arguing
that it significantly alleviates this challenge. Our approach yields two
notable advancements: first, the ability to assimilate new information without
necessitating alterations to the current policy, and second, the capability to
replay experiences without succumbing to policy divergence. Furthermore, when
contrasted with other experience replay (ER) techniques, our method
demonstrates the added advantage of being computationally efficient in an
online setting. We demonstrate that the proposed methodology enhances the
efficiency of learning, showcasing its potential impact on neuromorphic and
real-world applications.

强化学习面临着应用于现实问题的巨大挑战，主要源于有限的与环境交互导致的可用数据的稀缺性。本研究引入了生物学上可行的近端策略优化的实现，通过在重要领域中显著减轻这一挑战，提高了学习的效率。

在脉冲神经网络中学习快速变化的缓慢

Learning fast changing slow in spiking neural networks

Reinforcement Learning(RL) has achieved tremendous development in recent
years, but still faces significant obstacles in addressing complex real-life
problems due to the issues of poor system generalization, low sample efficiency
as well as safety and interpretability concerns. The core reason underlying
such dilemmas can be attributed to the fact that most of the work has focused
on the computational aspect of value functions or policies using a
representational model to describe atomic components of rewards, states and
actions etc, thus neglecting the rich high-level declarative domain knowledge
of facts, relations and rules that can be either provided a priori or acquired
through reasoning over time. Recently, there has been a rapidly growing
interest in the use of Knowledge Representation and Reasoning(KRR) methods,
usually using logical languages, to enable more abstract representation and
efficient learning in RL. In this survey, we provide a preliminary overview on
these endeavors that leverage the strengths of KRR to help solving various
problems in RL, and discuss the challenging open problems and possible
directions for future work in this area.

该文综述了利用知识表示与推理方法在强化学习中更抽象表现和更有效学习的领域中取得的进展和应用，并讨论了未来在此领域中需要解决的挑战和可能的方向。