Large Language Model (LLM) pre-training exhausts an ever growing compute
budget, yet recent research has demonstrated that careful document selection
enables comparable model quality with only a fraction of the FLOPs. Inspired by
efforts suggesting that domain-specific training document selection is in fact
an interpretable process [Gunasekar et al., 2023], as well as research showing
that instruction-finetuned LLMs are adept zero-shot data labelers [Gilardi et
al.,2023], we explore a promising direction for scalable general-domain
document selection; employing a prompted LLM as a document grader, we distill
quality labels into a classifier model, which is applied at scale to a large,
and already heavily-filtered, web-crawl-derived corpus autonomously. Following
the guidance of this classifier, we drop 75% of the corpus and train LLMs on
the remaining data. Results across multiple benchmarks show that: 1. Filtering
allows us to quality-match a model trained on the full corpus across diverse
benchmarks with at most 70% of the FLOPs, 2. More capable LLM labelers and
classifier models lead to better results that are less sensitive to the
labeler's prompt, 3. In-context learning helps to boost the performance of
less-capable labeling models. In all cases we use open-source datasets, models,
recipes, and evaluation frameworks, so that results can be reproduced by the
community.

LLM 预训练模型通过精心选择文档，可以在仅使用部分 FLOPs 的情况下实现与完整训练相当的模型质量；通过使用提示的 LLM 作为文档评分器，我们将质量标签提取并应用于大规模的互联网抓取数据集，以自动筛除部分文档，从而更好地匹配性能，并通过在上下文中学习来提高标签模型的性能。

大规模语言模型引导的文档选择

Large Language Model-guided Document Selection

Emotion recognition in text, the task of identifying emotions such as joy or
anger, is a challenging problem in NLP with many applications. One of the
challenges is the shortage of available datasets that have been annotated with
emotions. Certain existing datasets are small, follow different emotion
taxonomies and display imbalance in their emotion distribution. In this work,
we studied the impact of data augmentation techniques precisely when applied to
small imbalanced datasets, for which current state-of-the-art models (such as
RoBERTa) under-perform. Specifically, we utilized four data augmentation
methods (Easy Data Augmentation EDA, static and contextual Embedding-based, and
ProtAugment) on three datasets that come from different sources and vary in
size, emotion categories and distributions. Our experimental results show that
using the augmented data when training the classifier model leads to
significant improvements. Finally, we conducted two case studies: a) directly
using the popular chat-GPT API to paraphrase text using different prompts, and
b) using external data to augment the training set. Results show the promising
potential of these methods.

情感识别在文本中，识别诸如喜悦或愤怒之类的情感，是 NLP 中的一个具有挑战性的问题，其中一个挑战是缺乏已注释有情感的可用数据集。本研究研究了数据增强技术对小规模不平衡数据集的影响，并且证实使用增强数据训练分类器模型可以显著改进情感识别性能。