Reinforcement learning (RL) is a flexible and efficient method for
programming micro-robots in complex environments. Here we investigate whether
reinforcement learning can provide insights into biological systems when
trained to perform chemotaxis. Namely, whether we can learn about how
intelligent agents process given information in order to swim towards a target.
We run simulations covering a range of agent shapes, sizes, and swim speeds to
determine if the physical constraints on biological swimmers, namely Brownian
motion, lead to regions where reinforcement learners' training fails. We find
that the RL agents can perform chemotaxis as soon as it is physically possible
and, in some cases, even before the active swimming overpowers the stochastic
environment. We study the efficiency of the emergent policy and identify
convergence in agent size and swim speeds. Finally, we study the strategy
adopted by the reinforcement learning algorithm to explain how the agents
perform their tasks. To this end, we identify three emerging dominant
strategies and several rare approaches taken. These strategies, whilst
producing almost identical trajectories in simulation, are distinct and give
insight into the possible mechanisms behind which biological agents explore
their environment and respond to changing conditions.

探究强化学习在生物系统中的应用，以了解智能体的信息处理方式，研究其在趋化性游动中的应用和策略。

多智能体强化学习中的化学趋化策略的出现

Emergence of Chemotactic Strategies with Multi-Agent Reinforcement  Learning

We investigate the boundary between chemotaxis driven by spatial estimation
of gradients and chemotaxis driven by temporal estimation. While it is well
known that spatial chemotaxis becomes disadvantageous for small organisms at
high noise levels, it is unclear whether there is a discontinuous switch of
optimal strategies or a continuous transition exists. Here, we employ deep
reinforcement learning to study the possible integration of spatial and
temporal information in an a priori unconstrained manner. We parameterize such
a combined chemotactic policy by a recurrent neural network and evaluate it
using a minimal theoretical model of a chemotactic cell. By comparing with
constrained variants of the policy, we show that it converges to purely
temporal and spatial strategies at small and large cell sizes, respectively. We
find that the transition between the regimes is continuous, with the combined
strategy outperforming in the transition region both the constrained variants
as well as models that explicitly integrate spatial and temporal information.
Finally, by utilizing the attribution method of integrated gradients, we show
that the policy relies on a non-trivial combination of spatially and temporally
derived gradient information in a ratio that varies dynamically during the
chemotactic trajectories.

本研究通过深度强化学习方法，研究了在化学趋向作用中由空间梯度估计和时间梯度估计驱动的边界，并发现混合化学趋向策略在小细胞和大细胞之间的过渡是连续的，并且在过渡区域，混合化学趋向策略优于具有约束性的变体以及明确整合空间和时间信息的模型。

学习噪声趋化中空间和时间信息的最优集成

Learning optimal integration of spatial and temporal information in  noisy chemotaxis

The coupled problem of hydrodynamics and solute transport for the
Najafi-Golestanian three-sphere swimmer is studied, with the Reynolds number
set to zero and P\'eclet numbers (Pe) ranging from 0.06 to 60. The adopted
method is the numerical simulation of the problem with a finite element code
based upon the FEniCS library. For the swimmer executing the optimal locomotion
gait, we report the Sherwood number as a function of Pe in homogeneous fluids
and confirm that little gain in solute flux is achieved by swimming unless Pe
is significantly larger than 10. We also consider the swimmer as an learning
agent moving inside a fluid that has a concentration gradient. The outcomes of
Q-learning processes show that learning locomotion (with the displacement as
reward) is significantly easier than learning chemotaxis (with the increase of
solute flux as reward). The chemotaxis problem, even at low Pe, has a varying
environment that renders learning more difficult. Further, the learning
difficulty increases severely with the P\'eclet number. The results demonstrate
the challenges that natural and artificial swimmers need to overcome to migrate
efficiently when exposed to chemical inhomogeneities.

本文研究了从 Reynolds 数为零到 Péclet 数（Pe）在 0.06 至 60 范围内的情况下，涉及纳贾菲 - 戈勒斯塔尼安三球游泳者的流体动力学和溶质传输的耦合问题，并采用基于 FEniCS 库的有限元代码对问题进行了数值模拟。研究结果表明，在均质流体中，只有在 Pe 显著大于 10 时，游泳可以获得化学物质通量的较小增益。此外，研究发现，对于涉及溶质梯度的游泳者，学习定位会比学习趋化更容易，尤其是在 Pe 较低的情况下趋化的难度将显著增加。