For years the model performance in machine learning obeyed a power-law
relationship with the model size. For the consideration of parameter
efficiency, recent studies focus on increasing model depth rather than width to
achieve better performance. In this paper, we study how model width affects the
Transformer model through a parameter-efficient multi-path structure. To better
fuse features extracted from different paths, we add three additional
operations to each sublayer: a normalization at the end of each path, a cheap
operation to produce more features, and a learnable weighted mechanism to fuse
all features flexibly. Extensive experiments on 12 WMT machine translation
tasks show that, with the same number of parameters, the shallower multi-path
model can achieve similar or even better performance than the deeper model. It
reveals that we should pay more attention to the multi-path structure, and
there should be a balance between the model depth and width to train a better
large-scale Transformer.

本文研究了通过参数效率的多路径结构，如何影响变压器 (Transformer) 模型，并通过 12 个 WMT 机器翻译任务的广泛实验表明，使用相同数量的参数，较浅的多路径模型可以实现类似甚至更好的性能，揭示了在训练更好的大型 Transformer 时，应该注意多路径结构和模型深度和宽度之间的平衡。

多路径变换器更好：神经机器翻译的案例研究

Multi-Path Transformer is Better: A Case Study on Neural Machine  Translation

Transformers have demonstrated great potential in computer vision tasks. To
avoid dense computations of self-attentions in high-resolution visual data,
some recent Transformer models adopt a hierarchical design, where
self-attentions are only computed within local windows. This design
significantly improves the efficiency but lacks global feature reasoning in
early stages. In this work, we design a multi-path structure of the
Transformer, which enables local-to-global reasoning at multiple granularities
in each stage. The proposed framework is computationally efficient and highly
effective. With a marginal increasement in computational overhead, our model
achieves notable improvements in both image classification and semantic
segmentation. Code is available at this https URL

本研究提出了多通路结构的 Transformer 模型，实现局部到全局的多粒度特征推理，相较于现有的分层设计模型，在增加了极小的计算量的同时，在图像分类和语义分割任务上取得了显著的提高。

局部到全局的视觉 Transformer 自注意力

Local-to-Global Self-Attention in Vision Transformers

Domain Adaptation (DA) attempts to transfer knowledge learned in the labeled
source domain to the unlabeled but related target domain without requiring
large amounts of target supervision. Recent advances in DA mainly proceed by
aligning the source and target distributions. Despite the significant success,
the adaptation performance still degrades accordingly when the source and
target domains encounter a large distribution discrepancy. We consider this
limitation may attribute to the insufficient exploration of domain-specialized
features because most studies merely concentrate on domain-general feature
learning in task-specific layers and integrate totally-shared convolutional
networks (convnets) to generate common features for both domains. In this
paper, we relax the completely-shared convnets assumption adopted by previous
DA methods and propose Domain Conditioned Adaptation Network (DCAN), which
introduces domain conditioned channel attention module with a multi-path
structure to separately excite channel activation for each domain. Such a
partially-shared convnets module allows domain-specialized features in
low-level to be explored appropriately. Further, given the knowledge
transferability varying along with convolutional layers, we develop Generalized
Domain Conditioned Adaptation Network (GDCAN) to automatically determine
whether domain channel activations should be separately modeled in each
attention module. Afterward, the critical domain-specialized knowledge could be
adaptively extracted according to the domain statistic gaps. As far as we know,
this is the first work to explore the domain-wise convolutional channel
activations separately for deep DA networks. Additionally, to effectively match
high-level feature distributions across domains, we consider deploying feature
adaptation blocks after task-specific layers, which can explicitly mitigate the
domain discrepancy.

本文提出了一种基于多通路结构、利用领域专业特征进行自适应知识提取的域适应方法，通过添加特定领域的注意力机制探索低层次的领域特异性特征以及特定层次的知识适应块来提高适应性。