We introduce ChatGLM, an evolving family of large language models that we
have been developing over time. This report primarily focuses on the GLM-4
language series, which includes GLM-4, GLM-4-Air, and GLM-4-9B. They represent
our most capable models that are trained with all the insights and lessons
gained from the preceding three generations of ChatGLM. To date, the GLM-4
models are pre-trained on ten trillions of tokens mostly in Chinese and
English, along with a small set of corpus from 24 languages, and aligned
primarily for Chinese and English usage. The high-quality alignment is achieved
via a multi-stage post-training process, which involves supervised fine-tuning
and learning from human feedback. Evaluations show that GLM-4 1) closely rivals
or outperforms GPT-4 in terms of general metrics such as MMLU, GSM8K, MATH,
BBH, GPQA, and HumanEval, 2) gets close to GPT-4-Turbo in instruction following
as measured by IFEval, 3) matches GPT-4 Turbo (128K) and Claude 3 for long
context tasks, and 4) outperforms GPT-4 in Chinese alignments as measured by
AlignBench. The GLM-4 All Tools model is further aligned to understand user
intent and autonomously decide when and which tool(s) touse -- including web
browser, Python interpreter, text-to-image model, and user-defined functions --
to effectively complete complex tasks. In practical applications, it matches
and even surpasses GPT-4 All Tools in tasks like accessing online information
via web browsing and solving math problems using Python interpreter. Over the
course, we have open-sourced a series of models, including ChatGLM-6B (three
generations), GLM-4-9B (128K, 1M), GLM-4V-9B, WebGLM, and CodeGeeX, attracting
over 10 million downloads on Hugging face in the year 2023 alone. The open
models can be accessed through this https URL and
this https URL

介绍了 ChatGLM 的发展历程以及 GLM-4 系列语言模型的特点和优势，包括它们通过多阶段的预训练过程实现高质量的对齐，并且在多项评估中表现出与 GPT-4 相媲美甚至超越的性能，同时还介绍了一系列开源模型所取得的成功成果。

ChatGLM: 从 GLM-130B 到 GLM-4 的大型语言模型工具家族

ChatGLM: A Family of Large Language Models from GLM-130B to GLM-4 All  Tools

This study aims to innovatively explore adaptive applications of large
language models (LLM) in urban renewal. It also aims to improve its performance
and text generation quality for knowledge question-answering (QA) tasks. Based
on the ChatGLM, we automatically generate QA datasets using urban renewal
scientific literature corpora in a self-instruct manner and then conduct joint
fine-tuning training on the model using the Prefix and LoRA fine-tuning methods
to create an LLM for urban renewal. By guiding the LLM to automatically
generate QA data based on prompt words and given text, it is possible to
quickly obtain datasets in the urban renewal field and provide data support for
the fine-tuning training of LLMs. The experimental results show that the joint
fine-tuning training method proposed in this study can significantly improve
the performance of LLM on the QA tasks. Compared with LoRA fine-tuning, the
method improves the Bleu and Rouge metrics on the test by about 5%; compared
with the model before fine-tuning, the method improves the Bleu and Rouge
metrics by about 15%-20%. This study demonstrates the effectiveness and
superiority of the joint fine-tuning method using Prefix and LoRA for ChatGLM
in the urban renewal knowledge QA tasks. It provides a new approach for
fine-tuning LLMs on urban renewal-related tasks.

通过在城市更新领域中自动产生问答数据集，并使用 Prefix 和 LoRA 的联合微调方法对模型进行联合微调训练，本研究提出的联合微调训练方法能够显著提高 LLM 在知识问答任务上的性能，相比 LoRA 微调，该方法使测试中的 Bleu 和 Rouge 指标提高了约 5%；相比微调前的模型，该方法使 Bleu 和 Rouge 指标提高了约 15% -20%。该研究证明了在城市更新知识问答任务中使用 Prefix 和 LoRA 的联合微调方法对 ChatGLM 的有效性和优越性，为 LLM 在城市更新相关任务上的微调提供了新的方法。