State-of-the-art systems neuroscience experiments yield large-scale
multimodal data, and these data sets require new tools for analysis. Inspired
by the success of large pretrained models in vision and language domains, we
reframe the analysis of large-scale, cellular-resolution neuronal spiking data
into an autoregressive spatiotemporal generation problem. Neuroformer is a
multimodal, multitask generative pretrained transformer (GPT) model that is
specifically designed to handle the intricacies of data in systems
neuroscience. It scales linearly with feature size, can process an arbitrary
number of modalities, and is adaptable to downstream tasks, such as predicting
behavior. We first trained Neuroformer on simulated datasets, and found that it
both accurately predicted simulated neuronal circuit activity, and also
intrinsically inferred the underlying neural circuit connectivity, including
direction. When pretrained to decode neural responses, the model predicted the
behavior of a mouse with only few-shot fine-tuning, suggesting that the model
begins learning how to do so directly from the neural representations
themselves, without any explicit supervision. We used an ablation study to show
that joint training on neuronal responses and behavior boosted performance,
highlighting the model's ability to associate behavioral and neural
representations in an unsupervised manner. These findings show that Neuroformer
can analyze neural datasets and their emergent properties, informing the
development of models and hypotheses associated with the brain.

神经形态学数据的生成预训练变换器模型 Neuroformer 能分析神经数据集及其出现的属性，并推动与大脑相关的模型和假设的发展。

神经形态学：用于脑数据的多模态和多任务生成预训练

Neuroformer: Multimodal and Multitask Generative Pretraining for Brain  Data

In theoretical neuroscience, recent work leverages deep learning tools to
explore how some network attributes critically influence its learning dynamics.
Notably, initial weight distributions with small (resp. large) variance may
yield a rich (resp. lazy) regime, where significant (resp. minor) changes to
network states and representation are observed over the course of learning.
However, in biology, neural circuit connectivity generally has a low-rank
structure and therefore differs markedly from the random initializations
generally used for these studies. As such, here we investigate how the
structure of the initial weights, in particular their effective rank,
influences the network learning regime. Through both empirical and theoretical
analyses, we discover that high-rank initializations typically yield smaller
network changes indicative of lazier learning, a finding we also confirm with
experimentally-driven initial connectivity in recurrent neural networks.
Conversely, low-rank initialization biases learning towards richer learning.
Importantly, however, as an exception to this rule, we find lazier learning can
still occur with a low-rank initialization that aligns with task and data
statistics. Our research highlights the pivotal role of initial weight
structures in shaping learning regimes, with implications for metabolic costs
of plasticity and risks of catastrophic forgetting.

通过理论和实证分析，我们研究发现，初始权重的结构对网络学习机制起着重要影响，具有高秩的初始权重表现出更懒惰的学习，低秩的初始权重倾向于更丰富的学习，同时我们还发现与任务和数据统计信息一致的低秩初始权重仍然可能导致懒散学习。这一研究突出了初始权重结构在塑性代谢成本和灾难性遗忘风险方面的关键作用。