Large language models have been shown to achieve remarkable performance
across a variety of natural language tasks using few-shot learning, which
drastically reduces the number of task-specific training examples needed to
adapt the model to a particular application. To further our understanding of
the impact of scale on few-shot learning, we trained a 540-billion parameter,
densely activated, Transformer language model, which we call Pathways Language
Model PaLM. We trained PaLM on 6144 TPU v4 chips using Pathways, a new ML
system which enables highly efficient training across multiple TPU Pods. We
demonstrate continued benefits of scaling by achieving state-of-the-art
few-shot learning results on hundreds of language understanding and generation
benchmarks. On a number of these tasks, PaLM 540B achieves breakthrough
performance, outperforming the finetuned state-of-the-art on a suite of
multi-step reasoning tasks, and outperforming average human performance on the
recently released BIG-bench benchmark. A significant number of BIG-bench tasks
showed discontinuous improvements from model scale, meaning that performance
steeply increased as we scaled to our largest model. PaLM also has strong
capabilities in multilingual tasks and source code generation, which we
demonstrate on a wide array of benchmarks. We additionally provide a
comprehensive analysis on bias and toxicity, and study the extent of training
data memorization with respect to model scale. Finally, we discuss the ethical
considerations related to large language models and discuss potential
mitigation strategies.

通过使用 Pathways ML 系统训练 PaLM，我们证明了模型规模对少样本学习的影响，PaLM 540B 在大量语言理解、生成基准测试中取得了突破性表现，并在多步推理任务中超越了微调表现，另外还有强大的多语种任务和源代码生成能力，以及对偏差和毒性的全面分析。

PaLM: 基于路径的语言建模规模化

PaLM: Scaling Language Modeling with Pathways

This paper studies the relative importance of attention heads in
Transformer-based models to aid their interpretability in cross-lingual and
multi-lingual tasks. Prior research has found that only a few attention heads
are important in each mono-lingual Natural Language Processing (NLP) task and
pruning the remaining heads leads to comparable or improved performance of the
model. However, the impact of pruning attention heads is not yet clear in
cross-lingual and multi-lingual tasks. Through extensive experiments, we show
that (1) pruning a number of attention heads in a multi-lingual
Transformer-based model has, in general, positive effects on its performance in
cross-lingual and multi-lingual tasks and (2) the attention heads to be pruned
can be ranked using gradients and identified with a few trial experiments. Our
experiments focus on sequence labeling tasks, with potential applicability on
other cross-lingual and multi-lingual tasks. For comprehensiveness, we examine
two pre-trained multi-lingual models, namely multi-lingual BERT (mBERT) and
XLM-R, on three tasks across 9 languages each. We also discuss the validity of
our findings and their extensibility to truly resource-scarce languages and
other task settings.

该论文研究了注意力头在 Transformer 模型中的相对重要性，以帮助其在跨语言和多语言任务中的可解释性。通过大量实验，证明了在多语言 Transformer 模型中修剪注意力头可以在跨语言和多语言任务中提高模型性能，并且可以使用梯度对注意头进行排序和识别。

Transformer 注意力头在多语言和跨语言任务中的贡献

Contributions of Transformer Attention Heads in Multi- and Cross-lingual  Tasks

Natural language processing (NLP) tasks (e.g. question-answering in English)
benefit from knowledge of other tasks (e.g. named entity recognition in
English) and knowledge of other languages (e.g. question-answering in Spanish).
Such shared representations are typically learned in isolation, either across
tasks or across languages. In this work, we propose a meta-learning approach to
learn the interactions between both tasks and languages. We also investigate
the role of different sampling strategies used during meta-learning. We present
experiments on five different tasks and six different languages from the XTREME
multilingual benchmark dataset. Our meta-learned model clearly improves in
performance compared to competitive baseline models that also include
multi-task baselines. We also present zero-shot evaluations on unseen target
languages to demonstrate the utility of our proposed model.

通过元学习方法，将任务和语言相互作用的表征共享，可提高自然语言处理任务和多语种任务性能。