Large language models (LLMs) have showcased profound capabilities in language
understanding and generation, facilitating a wide array of applications.
However, there is a notable paucity of detailed, open-sourced methodologies on
efficiently scaling LLMs beyond 50 billion parameters with minimum
trial-and-error cost and computational resources. In this report, we introduce
Tele-FLM (aka FLM-2), a 52B open-sourced multilingual large language model that
features a stable, efficient pre-training paradigm and enhanced factual
judgment capabilities. Tele-FLM demonstrates superior multilingual language
modeling abilities, measured by BPB on textual corpus. Besides, in both English
and Chinese foundation model evaluation, it is comparable to strong
open-sourced models that involve larger pre-training FLOPs, such as Llama2-70B
and DeepSeek-67B. In addition to the model weights, we share the core designs,
engineering practices, and training details, which we expect to benefit both
the academic and industrial communities.

大语言模型的扩展和效率提升方法的开放资源，以及展示优于其他模型的跨语言语言建模能力和核心设计。

Tele-FLM 技术报告

Tele-FLM Technical Report

A major consideration in multilingual language modeling is how to best
represent languages with diverse vocabularies and scripts. Although
contemporary text encoding methods cover most of the world's writing systems,
they exhibit bias towards the high-resource languages of the Global West. As a
result, texts of underrepresented languages tend to be segmented into long
sequences of linguistically meaningless units. To address the disparities, we
introduce a new paradigm that encodes the same information with segments of
consistent size across diverse languages. Our encoding convention (MYTE) is
based on morphemes, as their inventories are more balanced across languages
than characters, which are used in previous methods. We show that MYTE produces
shorter encodings for all 99 analyzed languages, with the most notable
improvements for non-European languages and non-Latin scripts. This, in turn,
improves multilingual LM performance and diminishes the perplexity gap
throughout diverse languages.

通过使用一种基于词素的编码约定（MYTE），我们能够代表多种语言的相同信息，并改善多语言语言模型性能以及减少了不同语言之间的困惑度差异。

MYTE: 基于形态的字节编码以改进和公平多语言建模

MYTE: Morphology-Driven Byte Encoding for Better and Fairer Multilingual  Language Modeling

Recent advances in multimodal vision and language modeling have predominantly
focused on the English language, mostly due to the lack of multilingual
multimodal datasets to steer modeling efforts. In this work, we address this
gap and provide xGQA, a new multilingual evaluation benchmark for the visual
question answering task. We extend the established English GQA dataset to 7
typologically diverse languages, enabling us to detect and explore crucial
challenges in cross-lingual visual question answering. We further propose new
adapter-based approaches to adapt multimodal transformer-based models to become
multilingual, and -- vice versa -- multilingual models to become multimodal.
Our proposed methods outperform current state-of-the-art multilingual
multimodal models (e.g., M3P) in zero-shot cross-lingual settings, but the
accuracy remains low across the board; a performance drop of around 38 accuracy
points in target languages showcases the difficulty of zero-shot cross-lingual
transfer for this task. Our results suggest that simple cross-lingual transfer
of multimodal models yields latent multilingual multimodal misalignment,
calling for more sophisticated methods for vision and multilingual language
modeling.

本文提出了 xGQA，一个用于跨语言视觉问答任务的新的多语言评估基准，并使用适配器方法将多模型变换器模型扩展为多语言模型，结果表明简单的跨语言模型转移会导致多语言多模态失配，需要更复杂的方法来进行跨语言视觉和多语言语言建模

跨语言视觉问答

xGQA: Cross-Lingual Visual Question Answering

Prior studies in multilingual language modeling (e.g., Cotterell et al.,
2018; Mielke et al., 2019) disagree on whether or not inflectional morphology
makes languages harder to model. We attempt to resolve the disagreement and
extend those studies. We compile a larger corpus of 145 Bible translations in
92 languages and a larger number of typological features. We fill in missing
typological data for several languages and consider corpus-based measures of
morphological complexity in addition to expert-produced typological features.
We find that several morphological measures are significantly associated with
higher surprisal when LSTM models are trained with BPE-segmented data. We also
investigate linguistically-motivated subword segmentation strategies like
Morfessor and Finite-State Transducers (FSTs) and find that these segmentation
strategies yield better performance and reduce the impact of a language's
morphology on language modeling.

本文通过对 145 个语言的 92 个圣经翻译及其更多的分类特征的分析，探讨了语言的屈折变化是否使得多语言语言建模更加困难，发现基于神经网络的语言建模方法可以通过采用 FST 等分词策略来进一步降低语言构型的影响。