Large language models (LLMs) demonstrate remarkable ability to comprehend,
reason, and generate following nature language instructions. However, the
development of LLMs has been primarily focused on high-resource languages, such
as English, thereby limiting their applicability and research in other
languages. Consequently, we present PolyLM, a multilingual LLM trained on 640
billion (B) tokens, avaliable in two model sizes: 1.7B and 13B. To enhance its
multilingual capabilities, we 1) integrate bilingual data into training data;
and 2) adopt a curriculum learning strategy that increases the proportion of
non-English data from 30% in the first stage to 60% in the final stage during
pre-training. Further, we propose a multilingual self-instruct method which
automatically generates 132.7K diverse multilingual instructions for model
fine-tuning. To assess the model's performance, we collect several existing
multilingual tasks, including multilingual understanding, question answering,
generation, and translation. Extensive experiments show that PolyLM surpasses
other open-source models such as LLaMA and BLOOM on multilingual tasks while
maintaining comparable performance in English. Our models, alone with the
instruction data and multilingual benchmark, are available at:
https://modelscope.cn/models/damo/nlp_polylm_13b_text_generation.

PolyLM 是一个多语言大型语言模型，经过课程学习和自我指导方法的预训练，可以在多种不同语言的多语言理解、问答、生成和翻译任务上获得比其他已有模型更好的表现。

PolyLM: 一个开源的多语种大语言模型

PolyLM: An Open Source Polyglot Large Language Model

To avoid the "meaning conflation deficiency" of word embeddings, a number of
models have aimed to embed individual word senses. These methods at one time
performed well on tasks such as word sense induction (WSI), but they have since
been overtaken by task-specific techniques which exploit contextualized
embeddings. However, sense embeddings and contextualization need not be
mutually exclusive. We introduce PolyLM, a method which formulates the task of
learning sense embeddings as a language modeling problem, allowing
contextualization techniques to be applied. PolyLM is based on two underlying
assumptions about word senses: firstly, that the probability of a word
occurring in a given context is equal to the sum of the probabilities of its
individual senses occurring; and secondly, that for a given occurrence of a
word, one of its senses tends to be much more plausible in the context than the
others. We evaluate PolyLM on WSI, showing that it performs considerably better
than previous sense embedding techniques, and matches the current
state-of-the-art specialized WSI method despite having six times fewer
parameters. Code and pre-trained models are available at
this https URL

PolyLM 是一种基于语言建模的词义嵌入方法，能够融合上下文化嵌入技术，通过对词义的建模及概率计算实现更加准确的词义感知，在词义感知任务上性能优于先前的方法，且只有现有方法参数数量的六分之一。