It has long been assumed that the sheer number of parameters in large
language models (LLMs) drives in-context learning (ICL) capabilities, enabling
remarkable performance improvements by leveraging task-specific demonstrations.
Challenging this hypothesis, we introduce DEEP-ICL, a novel task Definition
Enriched ExPert Ensembling methodology for ICL. DEEP-ICL explicitly extracts
task definitions from given demonstrations and generates responses through
learning task-specific examples. We argue that improvement from ICL does not
directly rely on model size, but essentially stems from understanding task
definitions and task-guided learning. Inspired by this, DEEP-ICL combines two
3B models with distinct roles (one for concluding task definitions and the
other for learning task demonstrations) and achieves comparable performance to
LLaMA2-13B. Furthermore, our framework outperforms conventional ICL by
overcoming pretraining sequence length limitations, by supporting unlimited
demonstrations. We contend that DEEP-ICL presents a novel alternative for
achieving efficient few-shot learning, extending beyond the conventional ICL.

DEEP-ICL 是一种新颖的任务定义增强的专家集成学习方法，通过从给定的演示中显式地提取任务定义并通过学习任务特定的示例生成回答，它旨在实现高效的少样本学习，超越传统的上下文学习方法。

DEEP-ICL: 在上下文学习中丰富定义的专家语言模型

DEEP-ICL: Definition-Enriched Experts for Language Model In-Context  Learning

Generative information retrieval (IR) has experienced substantial growth
across multiple research communities (e.g., information retrieval, computer
vision, natural language processing, and machine learning), and has been highly
visible in the popular press. Theoretical, empirical, and actual user-facing
products have been released that retrieve documents (via generation) or
directly generate answers given an input request. We would like to investigate
whether end-to-end generative models are just another trend or, as some claim,
a paradigm change for IR. This necessitates new metrics, theoretical grounding,
evaluation methods, task definitions, models, user interfaces, etc. The goal of
this workshop (this https URL) is to focus on previously
explored Generative IR techniques like document retrieval and direct Grounded
Answer Generation, while also offering a venue for the discussion and
exploration of how Generative IR can be applied to new domains like
recommendation systems, summarization, etc. The format of the workshop is
interactive, including roundtable and keynote sessions and tends to avoid the
one-sided dialogue of a mini-conference.

本文研究生成信息检索是否只是另一种趋势或者像有些人声称的那样，是检索的范式变革，需要新的度量标准，理论基础，评估方法，任务定义，模型，用户界面等来探讨。

第一届生成式信息检索研讨会 @SIGIR 2023

Gen-IR @ SIGIR 2023: The First Workshop on Generative Information  Retrieval

Large language models (LLMs) have shown impressive performance in following
natural language instructions to solve unseen tasks. However, it remains
unclear whether models truly understand task definitions and whether the
human-written definitions are optimal. In this paper, we systematically study
the role of task definitions in instruction learning. We first conduct an
ablation analysis informed by human annotations to understand which parts of a
task definition are most important, and find that model performance only drops
substantially when removing contents describing the task output, in particular
label information. Next, we propose an automatic algorithm to compress task
definitions to a minimal supporting set of tokens, and find that 60\% of tokens
can be removed while maintaining or even improving model performance. Based on
these results, we propose two strategies to help models better leverage task
instructions: (1) providing only key information for tasks in a common
structured format, and (2) adding a meta-tuning stage to help the model better
understand the definitions. With these two strategies, we achieve a 4.2 Rouge-L
improvement over 119 unseen test tasks.

该研究使用人工注释的消融分析方法研究任务定义在指令学习中的作用，并提出两种策略以帮助大型语言模型更好地利用任务指令，包括提供以常见结构化格式呈现关键信息的任务，并添加一个元调整阶段，以帮助模型更好地理解定义。应用这两种策略，在 119 个未见过的测试任务中取得了 4.2 Rouge-L 的提高。