We present Sailor, a family of open language models ranging from 0.5B to 7B
parameters, tailored for South-East Asian (SEA) languages. These models are
continually pre-trained from Qwen1.5, a great language model for multilingual
use cases. From Qwen1.5, Sailor models accept 200B to 400B tokens, primarily
covering the languages of English, Chinese, Vietnamese, Thai, Indonesian,
Malay, and Lao. The training leverages several techniques, including BPE
dropout for improving the model robustness, aggressive data cleaning and
deduplication, and small proxy models to optimize data mixture. Experimental
results on four typical tasks indicate that Sailor models demonstrate strong
performance across different benchmarks, including commonsense reasoning,
question answering, reading comprehension and examination. Embracing the
open-source spirit, we share our insights through this report to spark a wider
interest in developing large language models for multilingual use cases.

Sailor 是一个面向东南亚语言的一系列开放语言模型，包括从 0.5B 到 7B 参数的模型。通过从 Qwen1.5 继续预训练，Sailor 模型接受 200B 到 400B 个 token，主要涵盖英语、中文、越南语、泰语、印度尼西亚语、马来语和老挝语。实验结果显示，Sailor 模型在常识推理、问答、阅读理解和考试等多个基准任务上表现出强的性能，鼓励开发更多多语种大语言模型的兴趣。

Sailor：东南亚开放式语言模型

Sailor: Open Language Models for South-East Asia

Graph Neural Networks (GNNs) have achieved state-of-the-art performance in
representation learning for graphs recently. However, the effectiveness of
GNNs, which capitalize on the key operation of message propagation, highly
depends on the quality of the topology structure. Most of the graphs in
real-world scenarios follow a long-tailed distribution on their node degrees,
that is, a vast majority of the nodes in the graph are tail nodes with only a
few connected edges. GNNs produce inferior node representations for tail nodes
since they lack structural information. In the pursuit of promoting the
expressiveness of GNNs for tail nodes, we explore how the deficiency of
structural information deteriorates the performance of tail nodes and propose a
general Structural Augmentation based taIL nOde Representation learning
framework, dubbed as SAILOR, which can jointly learn to augment the graph
structure and extract more informative representations for tail nodes.
Extensive experiments on public benchmark datasets demonstrate that SAILOR can
significantly improve the tail node representations and outperform the
state-of-the-art baselines.

利用结构增强的基于长尾分布的图神经网络模型（SAILOR）可以改善尾节点表达并超越现有基准模型。