We introduce the Yi model family, a series of language and multimodal models that demonstrate strong multi-dimensional capabilities. The Yi model family is based on 6B and 34B pretrained language models, then we extend them to chat models, 200K long context models, depth-upscaled models, and vision-language models. Our base models achieve strong performance on a wide range of benchmarks like MMLU, and our finetuned chat models deliver strong human preference rate on major evaluation platforms like AlpacaEval and Chatbot Arena. Building upon our scalable super-computing infrastructure and the classical transformer architecture, we attribute the performance of Yi models primarily to its data quality resulting from our data-engineering efforts. For pretraining, we construct 3.1 trillion tokens of English and Chinese corpora using a cascaded data deduplication and quality filtering pipeline. For finetuning, we polish a small scale (less than 10K) instruction dataset over multiple iterations such that every single instance has been verified directly by our machine learning engineers. For vision-language, we combine the chat language model with a vision transformer encoder and train the model to align visual representations to the semantic space of the language model. We further extend the context length to 200K through lightweight continual pretraining and demonstrate strong needle-in-a-haystack retrieval performance. We show that extending the depth of the pretrained checkpoint through continual pretraining further improves performance. We believe that given our current results, continuing to scale up model parameters using thoroughly optimized data will lead to even stronger frontier models.

我们介绍了Yi模型系列，这是一系列展现出强大多维能力的语言和多模态模型。我们的基础模型在MMLU等多个基准测试中取得了强大的性能，我们的微调聊天模型在AlpacaEval和Chatbot Arena等主要评估平台上展现了强大的人类偏好率。在可扩展的超级计算基础设施和经典的Transformer架构的基础上，我们主要归因于Yi模型的性能是由于数据质量，这是我们数据工程努力的结果。我们相信，通过彻底优化数据并进一步扩大模型参数规模，将会产生更强大的前沿模型。

Yi：01.AI的开放基础模型