Brain encoding with neuroimaging data is an established analysis aimed at
predicting human brain activity directly from complex stimuli features such as
movie frames. Typically, these features are the latent space representation
from an artificial neural network, and the stimuli are image, audio, or text
inputs. Ridge regression is a popular prediction model for brain encoding due
to its good out-of-sample generalization performance. However, training a ridge
regression model can be highly time-consuming when dealing with large-scale
deep functional magnetic resonance imaging (fMRI) datasets that include many
space-time samples of brain activity. This paper evaluates different
parallelization techniques to reduce the training time of brain encoding with
ridge regression on the CNeuroMod Friends dataset, one of the largest deep fMRI
resource currently available. With multi-threading, our results show that the
Intel Math Kernel Library (MKL) significantly outperforms the OpenBLAS library,
being 1.9 times faster using 32 threads on a single machine. We then evaluated
the Dask multi-CPU implementation of ridge regression readily available in
scikit-learn (MultiOutput), and we proposed a new "batch" version of Dask
parallelization, motivated by a time complexity analysis. In line with our
theoretical analysis, MultiOutput parallelization was found to be impractical,
i.e., slower than multi-threading on a single machine. In contrast, the
Batch-MultiOutput regression scaled well across compute nodes and threads,
providing speed-ups of up to 33 times with 8 compute nodes and 32 threads
compared to a single-threaded scikit-learn execution. Batch parallelization
using Dask thus emerges as a scalable approach for brain encoding with ridge
regression on high-performance computing systems using scikit-learn and large
fMRI datasets.

评估不同的并行化技术以减少在 CNeuroMod Friends 数据集上使用 ridge 回归进行脑编码的训练时间，结果显示 Intel Math Kernel Library 在单个机器上使用 32 个线程时比 OpenBLAS 库快 1.9 倍。使用 Dask 进行批处理并行化的 Batch-MultiOutput 回归在计算节点和线程上扩展得很好，在 8 个计算节点和 32 个线程相对于单线程的情况下加速了 33 倍，这种方法在高性能计算系统上使用 scikit-learn 和大型 fMRI 数据集进行脑编码具有可扩展性。

大规模个体 fMRI 数据集中脑编码的岭回归扩展

Scaling up ridge regression for brain encoding in a massive individual  fMRI dataset

Parallelization techniques have become ubiquitous for accelerating inference
and training of deep neural networks. Despite this, several operations are
still performed in a sequential manner. For instance, the forward and backward
passes are executed layer-by-layer, and the output of diffusion models is
produced by applying a sequence of denoising steps. This sequential approach
results in a computational cost proportional to the number of steps involved,
presenting a potential bottleneck as the number of steps increases. In this
work, we introduce DeepPCR, a novel algorithm which parallelizes typically
sequential operations used in inference and training of neural networks.
DeepPCR is based on interpreting a sequence of $L$ steps as the solution of a
specific system of equations, which we recover using the Parallel Cyclic
Reduction algorithm. This reduces the complexity of computing the sequential
operations from $\mathcal{O}(L)$ to $\mathcal{O}(\log_2L)$, thus yielding a
speedup for large $L$. To verify the theoretical lower complexity of the
algorithm, and to identify regimes for speedup, we test the effectiveness of
DeepPCR in parallelizing the forward and backward pass in multi-layer
perceptrons, and reach speedups of up to $30\times$ for forward and $200\times$
for backward pass. We additionally showcase the flexibility of DeepPCR by
parallelizing training of ResNets with as many as 1024 layers, and generation
in diffusion models, enabling up to $7\times$ faster training and $11\times$
faster generation, respectively, when compared to the sequential approach.

提出了一种名为 DeepPCR 的算法，它通过并行化典型的顺序操作，将计算复杂度从 O (L) 降低到 O (log2L)，从而在加速神经网络的训练和推理过程中提供更高的速度。

DeepPCR: 神经网络中的并行化顺序操作

DeepPCR: Parallelizing Sequential Operations in Neural Networks

The Graph Convolutional Network (GCN) model and its variants are powerful
graph embedding tools for facilitating classification and clustering on graphs.
However, a major challenge is to reduce the complexity of layered GCNs and make
them parallelizable and scalable on very large graphs -- state-of the art
techniques are unable to achieve scalability without losing accuracy and
efficiency. In this paper, we propose novel parallelization techniques for
graph sampling-based GCNs that achieve superior scalable performance on very
large graphs without compromising accuracy. Specifically, our GCN guarantees
work-efficient training and produces order of magnitude savings in computation
and communication. To scale GCN training on tightly-coupled shared memory
systems, we develop parallelization strategies for the key steps in training:
For the graph sampling step, we exploit parallelism within and across multiple
sampling instances, and devise an efficient data structure for concurrent
accesses that provides theoretical guarantee of near-linear speedup with number
of processing units. For the feature propagation step within the sampled graph,
we improve cache utilization and reduce DRAM communication by data
partitioning. We prove that our partitioning strategy is a 2-approximation for
minimizing the communication time compared to the optimal strategy. We
demonstrate that our parallel graph embedding outperforms state-of-the-art
methods in scalability (with respect to number of processors, graph size and
GCN model size), efficiency and accuracy on several large datasets. On a
40-core Xeon platform, our parallel training achieves $64\times$ speedup (with
AVX) in the sampling step and $25\times$ speedup in the feature propagation
step, compared to the serial implementation, resulting in a net speedup of
$21\times$.

本文提出了并行化技术，为图采样 GCN 提供卓越的可扩展性性能，在不妥协准确性的情况下，在非常大的图上实现了优秀的性能。具体地，我们保证了 GCN 的高效训练，并在计算和通信方面节约了数量级的开销，同时，我们的并行图嵌入在可扩展性、效率和准确性方面优于现有的方法。