We present WebGLM, a web-enhanced question-answering system based on the
General Language Model (GLM). Its goal is to augment a pre-trained large
language model (LLM) with web search and retrieval capabilities while being
efficient for real-world deployments. To achieve this, we develop WebGLM with
strategies for the LLM-augmented retriever, bootstrapped generator, and human
preference-aware scorer. Specifically, we identify and address the limitations
of WebGPT (OpenAI), through which WebGLM is enabled with accuracy, efficiency,
and cost-effectiveness advantages. In addition, we propose systematic criteria
for evaluating web-enhanced QA systems. We conduct multi-dimensional human
evaluation and quantitative ablation studies, which suggest the outperformance
of the proposed WebGLM designs over existing systems. WebGLM with the
10-billion-parameter GLM (10B) is shown to perform better than the
similar-sized WebGPT (13B) and even comparably to WebGPT (175B) in human
evaluation. The code, demo, and data are at
https://github.com/THUDM/WebGLM.

WebGLM 是一种基于 General Language Model 的网络增强问答系统，结合了网页搜索和检索功能，经过多维度的人类评估和量化削减研究表明，在准确性、效率和成本效益方面优于现有系统。

WebGLM：面向高效网络增强人类偏好的问答系统

WebGLM: Towards An Efficient Web-Enhanced Question Answering System with  Human Preferences

The common practice for training commonsense models has gone
from-human-to-corpus-to-machine: humans author commonsense knowledge graphs in
order to train commonsense models. In this work, we investigate an alternative,
from-machine-to-corpus-to-machine: general language models author these
commonsense knowledge graphs to train commonsense models. Our study leads to a
new framework, Symbolic Knowledge Distillation. As with prior art in Knowledge
Distillation (Hinton et al., 2015), our approach uses larger models to teach
smaller models. A key difference is that we distill knowledge symbolically-as
text-in addition to the neural model. We also distill only one aspect-the
commonsense of a general language model teacher, allowing the student to be a
different type, a commonsense model. Altogether, we show that careful prompt
engineering and a separately trained critic model allow us to selectively
distill high-quality causal commonsense from GPT-3, a general language model.
Empirical results demonstrate that, for the first time, a human-authored
commonsense knowledge graph is surpassed by our automatically distilled variant
in all three criteria: quantity, quality, and diversity. In addition, it
results in a neural commonsense model that surpasses the teacher model's
commonsense capabilities despite its 100x smaller size. We apply this to the
ATOMIC resource, and share our new symbolic knowledge graph and commonsense
models.

研究一个新的框架，基于 Symbolic Knowledge Distillation，使用通用语言模型自动化地从 ATOMIC 资源中筛选高质量因果常识，并创造一种新的常识模型，其大小只有原始模型的 1/100，但功能优于其老师模型。

符号知识蒸馏：从通用语言模型到常识模型

Symbolic Knowledge Distillation: from General Language Models to  Commonsense Models

There have been various types of pretraining architectures including
autoencoding models (e.g., BERT), autoregressive models (e.g., GPT), and
encoder-decoder models (e.g., T5). However, none of the pretraining frameworks
performs the best for all tasks of three main categories including natural
language understanding (NLU), unconditional generation, and conditional
generation. We propose a General Language Model (GLM) based on autoregressive
blank infilling to address this challenge. GLM improves blank filling
pretraining by adding 2D positional encodings and allowing an arbitrary order
to predict spans, which results in performance gains over BERT and T5 on NLU
tasks. Meanwhile, GLM can be pretrained for different types of tasks by varying
the number and lengths of blanks. On a wide range of tasks across NLU,
conditional and unconditional generation, GLM outperforms BERT, T5, and GPT
given the same model sizes and data, and achieves the best performance from a
single pretrained model with 1.25x parameters of BERT Large , demonstrating its
generalizability to different downstream tasks.

本文提出了一种基于自回归的空白填充预训练方法称为 GLM，该方法通过添加 2D 位置编码和允许任意顺序来预测跨度来改进填空预训练。实验证明 GLM 在自然语言理解、有条件和无条件生成任务上的性能表现要好于 BERT、T5 和 GPT，预训练时还可以将空白的数量和长度改变以适应不同的任务类型。