Reinforcement learning has a rich history in neuroscience, from early work on
dopamine as a reward prediction error signal for temporal difference learning
(Schultz et al., 1997) to recent work suggesting that dopamine could implement
a form of 'distributional reinforcement learning' popularized in deep learning
(Dabney et al., 2020). Throughout this literature, there has been a tight link
between theoretical advances in reinforcement learning and neuroscientific
experiments and findings. As a result, the theories describing our experimental
data have become increasingly complex and difficult to navigate. In this
review, we cover the basic theory underlying classical work in reinforcement
learning and build up to an introductory overview of methods used in modern
deep reinforcement learning that have found applications in systems
neuroscience. We start with an overview of the reinforcement learning problem
and classical temporal difference algorithms, followed by a discussion of
'model-free' and 'model-based' reinforcement learning together with methods
such as DYNA and successor representations that fall in between these two
categories. Throughout these sections, we highlight the close parallels between
the machine learning methods and related work in both experimental and
theoretical neuroscience. We then provide an introduction to deep reinforcement
learning with examples of how these methods have been used to model different
learning phenomena in the systems neuroscience literature, such as
meta-reinforcement learning (Wang et al., 2018) and distributional
reinforcement learning (Dabney et al., 2020). Code that implements the methods
discussed in this work and generates the figures is also provided.

强化学习和神经科学之间存在紧密的联系，本论文通过回顾经典强化学习和介绍现代深度强化学习的方法，以及其在系统神经科学中的应用，阐述了这种联系。

神经科学中的强化学习简介

An introduction to reinforcement learning for neuroscience

State-of-the-art systems neuroscience experiments yield large-scale
multimodal data, and these data sets require new tools for analysis. Inspired
by the success of large pretrained models in vision and language domains, we
reframe the analysis of large-scale, cellular-resolution neuronal spiking data
into an autoregressive spatiotemporal generation problem. Neuroformer is a
multimodal, multitask generative pretrained transformer (GPT) model that is
specifically designed to handle the intricacies of data in systems
neuroscience. It scales linearly with feature size, can process an arbitrary
number of modalities, and is adaptable to downstream tasks, such as predicting
behavior. We first trained Neuroformer on simulated datasets, and found that it
both accurately predicted simulated neuronal circuit activity, and also
intrinsically inferred the underlying neural circuit connectivity, including
direction. When pretrained to decode neural responses, the model predicted the
behavior of a mouse with only few-shot fine-tuning, suggesting that the model
begins learning how to do so directly from the neural representations
themselves, without any explicit supervision. We used an ablation study to show
that joint training on neuronal responses and behavior boosted performance,
highlighting the model's ability to associate behavioral and neural
representations in an unsupervised manner. These findings show that Neuroformer
can analyze neural datasets and their emergent properties, informing the
development of models and hypotheses associated with the brain.

神经形态学数据的生成预训练变换器模型 Neuroformer 能分析神经数据集及其出现的属性，并推动与大脑相关的模型和假设的发展。

神经形态学：用于脑数据的多模态和多任务生成预训练

Neuroformer: Multimodal and Multitask Generative Pretraining for Brain  Data

Systems neuroscience relies on two complementary views of neural data,
characterized by single neuron tuning curves and analysis of population
activity. These two perspectives combine elegantly in neural latent variable
models that constrain the relationship between latent variables and neural
activity, modeled by simple tuning curve functions. This has recently been
demonstrated using Gaussian processes, with applications to realistic and
topologically relevant latent manifolds. Those and previous models, however,
missed crucial shared coding properties of neural populations. We propose
feature sharing across neural tuning curves which significantly improves
performance and helps optimization. We also propose a solution to the ensemble
detection problem, where different groups of neurons, i.e., ensembles, can be
modulated by different latent manifolds. Achieved through a soft clustering of
neurons during training, this allows for the separation of mixed neural
populations in an unsupervised manner. These innovations lead to more
interpretable models of neural population activity that train well and perform
better even on mixtures of complex latent manifolds. Finally, we apply our
method on a recently published grid cell dataset, and recover distinct
ensembles, infer toroidal latents and predict neural tuning curves in a single
integrated modeling framework.

本文提出了神经潜在变量模型中的特征共享及对集合检测问题的解决方法，最终应用在网格细胞数据集上，以单一的集成建模框架恢复不同集合，推断环形潜在变量并预测神经调谐曲线。

共享特征和划分集合揭示潜在流形的神经编码

Understanding Neural Coding on Latent Manifolds by Sharing Features and  Dividing Ensembles

Over the last several years, the use of machine learning (ML) in neuroscience
has been rapidly increasing. Here, we review ML's contributions, both realized
and potential, across several areas of systems neuroscience. We describe four
primary roles of ML within neuroscience: 1) creating solutions to engineering
problems, 2) identifying predictive variables, 3) setting benchmarks for simple
models of the brain, and 4) serving itself as a model for the brain. The
breadth and ease of its applicability suggests that machine learning should be
in the toolbox of most systems neuroscientists.

该研究回顾了近年来机器学习在神经科学领域中的四个主要方面，包括解决工程问题、识别预测变量、建立大脑简单模型的基准和作为大脑模型的一个例子。机器学习的广泛适用性和易用性表明它应该成为大多数系统神经科学家的工具箱之一。