We describe the University of Amsterdam Intelligent Data Engineering Lab
team's entry for the SemEval-2024 Task 6 competition. The SHROOM-INDElab system
builds on previous work on using prompt programming and in-context learning
with large language models (LLMs) to build classifiers for hallucination
detection, and extends that work through the incorporation of context-specific
definition of task, role, and target concept, and automated generation of
examples for use in a few-shot prompting approach. The resulting system
achieved fourth-best and sixth-best performance in the model-agnostic track and
model-aware tracks for Task 6, respectively, and evaluation using the
validation sets showed that the system's classification decisions were
consistent with those of the crowd-sourced human labellers. We further found
that a zero-shot approach provided better accuracy than a few-shot approach
using automatically generated examples. Code for the system described in this
paper is available on Github.

通过使用大型语言模型进行提示程序设计和上下文学习来构建大学智能数据工程实验室团队参与 SemEval-2024 任务 6 竞赛的 SHROOM-INDElab 系统，该系统在上下文特定任务、角色和目标概念的定义以及自动生成示例的基础上，扩展了对幻觉检测的分类器的先前工作，并在模型无关和模型感知的任务中获得了第四和第六的性能，与众包人工标注的验证集一致，发现零样本方法提供了比使用自动生成示例的 few-shot 方法更好的准确性。

SHROOM-INDElab 在 SemEval-2024 任务 6 中的零击中和少击中基于 LLM 的幻觉检测分类

SHROOM-INDElab at SemEval-2024 Task 6: Zero- and Few-Shot LLM-Based  Classification for Hallucination Detection

Prevailing methods for mapping large generative language models to supervised
tasks may fail to sufficiently probe models' novel capabilities. Using GPT-3 as
a case study, we show that 0-shot prompts can significantly outperform few-shot
prompts. We suggest that the function of few-shot examples in these cases is
better described as locating an already learned task rather than meta-learning.
This analysis motivates rethinking the role of prompts in controlling and
evaluating powerful language models. In this work, we discuss methods of prompt
programming, emphasizing the usefulness of considering prompts through the lens
of natural language. We explore techniques for exploiting the capacity of
narratives and cultural anchors to encode nuanced intentions and techniques for
encouraging deconstruction of a problem into components before producing a
verdict. Informed by this more encompassing theory of prompt programming, we
also introduce the idea of a metaprompt that seeds the model to generate its
own natural language prompts for a range of tasks. Finally, we discuss how
these more general methods of interacting with language models can be
incorporated into existing and future benchmarks and practical applications.

使用 GPT-3 模型分析得出少量样例引导更接近于已学习的任务，本文探讨使用自然语言编程来优化引导，提出元引导概念后能够生成更多的任务自然语言引导，展望这些方法如何应用于实践应用。