In this study, we present evidence suggesting that depthwise convolutional
kernels are effectively replicating the structural intricacies of the
biological receptive fields observed in the mammalian retina. We provide
analytics of trained kernels from various state-of-the-art models
substantiating this evidence. Inspired by this intriguing discovery, we propose
an initialization scheme that draws inspiration from the biological receptive
fields. Experimental analysis of the ImageNet dataset with multiple CNN
architectures featuring depthwise convolutions reveals a marked enhancement in
the accuracy of the learned model when initialized with biologically derived
weights. This underlies the potential for biologically inspired computational
models to further our understanding of vision processing systems and to improve
the efficacy of convolutional networks.

我们提供了支持的分析，显示出深度卷积核能够有效地复制哺乳动物视网膜中观察到的生物感受野的结构复杂性。在此基础上，我们提出了一个启发于生物感受野的初始化方案，并利用多个包含深度卷积的 CNN 架构在 ImageNet 数据集上进行实验分析，结果表明使用生物启发的权重初始化可以明显提高学习模型的准确性，这突显了生物启发型计算模型进一步推动视觉处理系统理解和改善卷积网络效果的潜力。

神经回响：深层卷积滤波器复制生物感受野

Neural Echos: Depthwise Convolutional Filters Replicate Biological  Receptive Fields

While the expressive power and computational capabilities of graph neural
networks (GNNs) have been theoretically studied, their optimization and
learning dynamics, in general, remain largely unexplored. Our study undertakes
the Graph Attention Network (GAT), a popular GNN architecture in which a node's
neighborhood aggregation is weighted by parameterized attention coefficients.
We derive a conservation law of GAT gradient flow dynamics, which explains why
a high portion of parameters in GATs with standard initialization struggle to
change during training. This effect is amplified in deeper GATs, which perform
significantly worse than their shallow counterparts. To alleviate this problem,
we devise an initialization scheme that balances the GAT network. Our approach
i) allows more effective propagation of gradients and in turn enables
trainability of deeper networks, and ii) attains a considerable speedup in
training and convergence time in comparison to the standard initialization. Our
main theorem serves as a stepping stone to studying the learning dynamics of
positive homogeneous models with attention mechanisms.

基于图神经网络 (Graph Neural Networks) 的优化和学习动态尚未被广泛探索，本研究通过分析图注意力网络 (Graph Attention Network) 提出的保守定律解释了标准初始化中参数变动困难的现象，并提出了一种初始化方案来平衡图注意力网络，使得深层网络更易于训练并加速收敛速度。

GAT 是否失衡？

Are GATs Out of Balance?

We answer the question: "Does local progress (on batches) imply global
progress (on the entire dataset) for mini-batch $k$-means?". Specifically, we
consider mini-batch $k$-means which terminates only when the improvement in the
quality of the clustering on the sampled batch is below some threshold.
Although at first glance it appears that this algorithm might execute
forever, we answer the above question in the affirmative and show that if the
batch is of size $\tilde{\Omega}((d/\epsilon)^2)$, it must terminate within
$O(d/\epsilon)$ iterations with high probability, where $d$ is the dimension of
the input, and $\epsilon$ is a threshold parameter for termination. This is
true regardless of how the centers are initialized. When the algorithm is
initialized with the $k$-means++ initialization scheme, it achieves an
approximation ratio of $O(\log k)$ (the same as the full-batch version).
Finally, we show the applicability of our results to the mini-batch $k$-means
algorithm implemented in the scikit-learn (sklearn) python library.

本文考虑小批量 $k$-means 聚类算法的可行性及效果，证明了在合适的批量大小情况下，算法会在有限的时间内收敛，而且 $k$-means++ 算法能够取得和全量数据一样的近似比例。此外，还将结果应用到了 Python 的 Scikit-learn（sklearn）库中。

Mini-batch k-means 在 O (d/ε) 次迭代内终止

Mini-batch $k$-means terminates within $O(d/ε)$ iterations

Many optimization methods for generating black-box adversarial examples have
been proposed, but the aspect of initializing said optimizers has not been
considered in much detail. We show that the choice of starting points is indeed
crucial, and that the performance of state-of-the-art attacks depends on it.
First, we discuss desirable properties of starting points for attacking image
classifiers, and how they can be chosen to increase query efficiency. Notably,
we find that simply copying small patches from other images is a valid
strategy. We then present an evaluation on ImageNet that clearly demonstrates
the effectiveness of this method: Our initialization scheme reduces the number
of queries required for a state-of-the-art Boundary Attack by 81%,
significantly outperforming previous results reported for targeted black-box
adversarial examples.

本文研究黑盒对抗样本攻击的优化方法，发现初始点选取的重要性，并提出了一种有效的初始化方案， 较之以前的方法可以减少 81% 的查询数量。