Recently, significant public efforts have been directed towards developing
low-cost models with capabilities akin to ChatGPT, thereby fostering the growth
of open-source conversational models. However, there remains a scarcity of
comprehensive and in-depth evaluations of these models' performance. In this
study, we examine the influence of training data factors, including quantity,
quality, and linguistic distribution, on model performance. Our analysis is
grounded in several publicly accessible, high-quality instruction datasets, as
well as our own Chinese multi-turn conversations. We assess various models
using a evaluation set of 1,000 samples, encompassing nine real-world
scenarios. Our goal is to supplement manual evaluations with quantitative
analyses, offering valuable insights for the continued advancement of
open-source chat models. Furthermore, to enhance the performance and training
and inference efficiency of models in the Chinese domain, we extend the
vocabulary of LLaMA - the model with the closest open-source performance to
proprietary language models like GPT-3 - and conduct secondary pre-training on
3.4B Chinese words. We make our model, data, as well as code publicly
available.

本研究旨在使用公开数据集结合自身汉语多轮对话中的数据进行分析，选取各种评估指标来评价各类开源聊天机器人的性能表现，并对 LLaMA 进行词汇扩展及 34 亿汉语单词的二次预训练，以期提升聊天机器人在中文领域的表现与效率，最后将模型、数据、代码进行公开发布。