Transformer requires a fixed number of layers and heads which makes them
inflexible to the complexity of individual samples and expensive in training
and inference. To address this, we propose a sample-based Dynamic Hierarchical
Transformer (DHT) model whose layers and heads can be dynamically configured
with single data samples via solving contextual bandit problems. To determine
the number of layers and heads, we use the Uniform Confidence Bound while we
deploy combinatorial Thompson Sampling in order to select specific head
combinations given their number. Different from previous work that focuses on
compressing trained networks for inference only, DHT is not only advantageous
for adaptively optimizing the underlying network architecture during training
but also has a flexible network for efficient inference. To the best of our
knowledge, this is the first comprehensive data-driven dynamic transformer
without any additional auxiliary neural networks that implement the dynamic
system. According to the experiment results, we achieve up to 74% computational
savings for both training and inference with a minimal loss of accuracy.

通过解决上下文赌博问题动态配置层数和头数，我们提出了一种基于样本的动态层级变压器模型（DHT），与先前关注于仅推理的训练网络压缩不同，DHT 不仅有优势用于训练期间自适应优化底层网络架构，而且具备高效推理的灵活网络结构，得到高达 74％的计算节省同时最小化准确性损失。

基于样例的动态分层 Transformer：通过上下文强化学习实现层和头的灵活性

Sample-based Dynamic Hierarchical Transformer with Layer and Head  Flexibility via Contextual Bandit

Graph Neural Networks (GNNs) have attracted considerable attention and have
emerged as a new promising paradigm to process graph-structured data. GNNs are
usually stacked to multiple layers and the node representations in each layer
are computed through propagating and aggregating the neighboring node features
with respect to the graph. By stacking to multiple layers, GNNs are able to
capture the long-range dependencies among the data on the graph and thus bring
performance improvements. To train a GNN with multiple layers effectively, some
normalization techniques (e.g., node-wise normalization, batch-wise
normalization) are necessary. However, the normalization techniques for GNNs
are highly task-relevant and different application tasks prefer to different
normalization techniques, which is hard to know in advance. To tackle this
deficiency, in this paper, we propose to learn graph normalization by
optimizing a weighted combination of normalization techniques at four different
levels, including node-wise normalization, adjacency-wise normalization,
graph-wise normalization, and batch-wise normalization, in which the
adjacency-wise normalization and the graph-wise normalization are newly
proposed in this paper to take into account the local structure and the global
structure on the graph, respectively. By learning the optimal weights, we are
able to automatically select a single best or a best combination of multiple
normalizations for a specific task. We conduct extensive experiments on
benchmark datasets for different tasks, including node classification, link
prediction, graph classification and graph regression, and confirm that the
learned graph normalization leads to competitive results and that the learned
weights suggest the appropriate normalization techniques for the specific task.
Source code is released here this https URL

本文提出了通过优化四个不同层面的加权组合来学习图归一化的方法，其中包括节点归一化、邻接归一化、图归一化和批归一化。在不同数据集和任务中，学到的权重可以选择最优的归一化技巧，实现竞争性结果。

图神经网络的图归一化学习

Learning Graph Normalization for Graph Neural Networks

A key challenge in designing convolutional network models is sizing them
appropriately. Many factors are involved in these decisions, including number
of layers, feature maps, kernel sizes, etc. Complicating this further is the
fact that each of these influence not only the numbers and dimensions of the
activation units, but also the total number of parameters. In this paper we
focus on assessing the independent contributions of three of these linked
variables: The numbers of layers, feature maps, and parameters. To accomplish
this, we employ a recursive convolutional network whose weights are tied
between layers; this allows us to vary each of the three factors in a
controlled setting. We find that while increasing the numbers of layers and
parameters each have clear benefit, the number of feature maps (and hence
dimensionality of the representation) appears ancillary, and finds most of its
benefit through the introduction of more weights. Our results (i) empirically
confirm the notion that adding layers alone increases computational power,
within the context of convolutional layers, and (ii) suggest that precise
sizing of convolutional feature map dimensions is itself of little concern;
more attention should be paid to the number of parameters in these layers
instead.

本文考察卷积神经网络中层数、特征图数量和参数数量对模型性能的影响，结果表明，增加层数和参数数量都有明显改善，而特征图数量（和表示的维度）对模型性能影响较小，更应关注这些层中的参数数量。