Large language models (LLMs) have achieved dramatic proficiency over NLP
tasks with normal length. Recently, multiple studies have committed to
extending the context length and enhancing the long text modeling capabilities
of LLMs. To comprehensively evaluate the long context ability of LLMs, we
propose BAMBOO, a multi-task long context benchmark. BAMBOO has been designed
with four principles: comprehensive capacity evaluation, avoidance of data
contamination, accurate automatic evaluation, and different length levels. It
consists of 10 datasets from 5 different long text understanding tasks, i.e.
question answering, hallucination detection, text sorting, language modeling,
and code completion, to cover core capacities and various domains of LLMs. We
conduct experiments with five long context models on BAMBOO and further discuss
four key research questions of long text. We also qualitatively analyze current
long context models and point out future directions for enhancing long text
modeling capacities. We release our data, prompts, and code at
this https URL

多个研究已致力于扩展 LLMs 的上下文长度和增强其对长文本建模的能力。本文提出了 BAMBOO，一个多任务长上下文基准，并使用 10 个数据集从 5 个不同的长文本理解任务对 LLMs 进行全面评估，涵盖了核心能力和各个领域。通过在 BAMBOO 上进行实验，我们对五个长上下文模型进行了讨论，提出了四个关键的研究问题，并对当前的长上下文模型进行了质量分析，指出了增强长文本建模能力的未来方向。

BAMBOO：评估大型语言模型的长文本建模能力的综合基准

BAMBOO: A Comprehensive Benchmark for Evaluating Long Text Modeling  Capacities of Large Language Models

The meanings of words and phrases depend not only on where they are used
(contexts) but also on who use them (writers). Pretrained language models
(PLMs) are powerful tools for capturing context, but they are typically
pretrained and fine-tuned for universal use across different writers. This
study aims to improve the accuracy of text understanding tasks by personalizing
the fine-tuning of PLMs for specific writers. We focus on a general setting
where only the plain text from target writers are available for
personalization. To avoid the cost of fine-tuning and storing multiple copies
of PLMs for different users, we exhaustively explore using writer-specific
prompts to personalize a unified PLM. Since the design and evaluation of these
prompts is an underdeveloped area, we introduce and compare different types of
prompts that are possible in our setting. To maximize the potential of
prompt-based personalized fine-tuning, we propose a personalized intermediate
learning based on masked language modeling to extract task-independent traits
of writers' text. Our experiments, using multiple tasks, datasets, and PLMs,
reveal the nature of different prompts and the effectiveness of our
intermediate learning approach.

利用个性化中间学习提取作家文本的任务无关特征，通过使用作家特定提示来个性化统一的预训练语言模型，提高文本理解任务的准确性。

PerPLM：通过特定作家中间学习和提示个性化微调预训练语言模型

PerPLM: Personalized Fine-tuning of Pretrained Language Models via  Writer-specific Intermediate Learning and Prompts

Based on large-scale pre-trained multilingual representations, recent
cross-lingual transfer methods have achieved impressive transfer performances.
However, the performance of target languages still lags far behind the source
language. In this paper, our analyses indicate such a performance gap is
strongly associated with the cross-lingual representation discrepancy. To
achieve better cross-lingual transfer performance, we propose the cross-lingual
manifold mixup (X-Mixup) method, which adaptively calibrates the representation
discrepancy and gives a compromised representation for target languages.
Experiments on the XTREME benchmark show X-Mixup achieves 1.8% performance
gains on multiple text understanding tasks, compared with strong baselines, and
significantly reduces the cross-lingual representation discrepancy.

通过跨语言流域混合的 X-Mixup 方法，根据大规模的预训练的多语言表示，以解决不同语言之间表示差异的问题并提高多个文本理解任务的表现