We present our work on predicting United Nations sustainable development
goals (SDG) for university courses. We use an LLM named PaLM 2 to generate
training data given a noisy human-authored course description input as input.
We use this data to train several different smaller language models to predict
SDGs for university courses. This work contributes to better university level
adaptation of SDGs. The best performing model in our experiments was BART with
an F1-score of 0.786.

我们用一个名为 PaLM 2 的 LLM 来预测大学课程的联合国可持续发展目标（SDG），并使用它生成训练数据，基于一个嘈杂的人工撰写的课程描述作为输入。我们使用这些数据来训练几个不同的较小语言模型来预测大学课程的 SDGs。该研究有助于更好地使可持续发展目标在大学层面得到应用。在我们的实验中表现最佳的模型是 BART，其 F1 得分为 0.786。

利用课程描述预测可持续发展目标 —— 从 LLMs 到传统基础模型

Predicting Sustainable Development Goals Using Course Descriptions --  from LLMs to Conventional Foundation Models

We find that the best publicly available LLMs like GPT-4 and PaLM 2 currently
perform poorly at basic text handling required of lawyers or paralegals, such
as looking up the text at a line of a witness deposition or at a subsection of
a contract. We introduce a benchmark to quantify this poor performance, which
casts into doubt LLMs' current reliability as-is for legal practice. Finetuning
for these tasks brings an older LLM to near-perfect performance on our test set
and also raises performance on a related legal task. This stark result
highlights the need for more domain expertise in LLM training.

公开可获得的最好的 LLM（如 GPT-4 和 PaLM 2）在律师或律师助理所需的基本文本处理方面表现不佳，我们引入了一个基准来量化这种不良表现，这对于目前的 LLMs 在法律实践中的可靠性提出了疑问。为这些任务进行微调使得一种旧的 LLM 在我们的测试集上接近完美的表现，并提高了与法律相关的任务的表现。这个鲜明的结果凸显了在 LLM 训练中需要更多的领域专业知识。

BLT：大型语言模型能应对基础法律文本吗？

BLT: Can Large Language Models Handle Basic Legal Text?

We introduce PaLM 2, a new state-of-the-art language model that has better
multilingual and reasoning capabilities and is more compute-efficient than its
predecessor PaLM. PaLM 2 is a Transformer-based model trained using a mixture
of objectives. Through extensive evaluations on English and multilingual
language, and reasoning tasks, we demonstrate that PaLM 2 has significantly
improved quality on downstream tasks across different model sizes, while
simultaneously exhibiting faster and more efficient inference compared to PaLM.
This improved efficiency enables broader deployment while also allowing the
model to respond faster, for a more natural pace of interaction. PaLM 2
demonstrates robust reasoning capabilities exemplified by large improvements
over PaLM on BIG-Bench and other reasoning tasks. PaLM 2 exhibits stable
performance on a suite of responsible AI evaluations, and enables
inference-time control over toxicity without additional overhead or impact on
other capabilities. Overall, PaLM 2 achieves state-of-the-art performance
across a diverse set of tasks and capabilities.
When discussing the PaLM 2 family, it is important to distinguish between
pre-trained models (of various sizes), fine-tuned variants of these models, and
the user-facing products that use these models. In particular, user-facing
products typically include additional pre- and post-processing steps.
Additionally, the underlying models may evolve over time. Therefore, one should
not expect the performance of user-facing products to exactly match the results
reported in this report.

PaLM 2 是个新的最先进的语言模型，比其前身 PaLM 在多语言和推理能力方面更具优势，同时也更加计算效率高。通过在英语、多语言和推理任务上的全面评估，我们证明了 PaLM 2 在不同模型大小的下游任务上的质量显著提高，同时推理速度更快、更高效，这提高了交互的自然节奏。因此，PaLM 2 达到了各种任务和能力的最先进水平。