Since large language models have approached human-level performance on many
tasks, it has become increasingly harder for researchers to find tasks that are
still challenging to the models. Failure cases usually come from the long-tail
distribution - data that an oracle language model could assign a probability on
the lower end of its distribution. Current methodology such as prompt
engineering or crowdsourcing are insufficient for creating long-tail examples
because humans are constrained by cognitive bias. We propose a
Logic-Induced-Knowledge-Search (LINK) framework for systematically generating
long-tail knowledge statements. Grounded by a symbolic rule, we search for
long-tail values for each variable of the rule by first prompting a LLM, then
verifying the correctness of the values with a critic, and lastly pushing for
the long-tail distribution with a reranker. With this framework we construct a
dataset, Logic-Induced-Long-Tail (LINT), consisting of 200 symbolic rules and
50K knowledge statements spanning across four domains. Human annotations find
that 84% of the statements in LINT are factually correct. In contrast, ChatGPT
and GPT4 struggle with directly generating long-tail statements under the
guidance of logic rules, each only getting 56% and 78% of their statements
correct. Moreover, their "long-tail" generations in fact fall into the higher
likelihood range, and thus are not really long-tail. Our findings suggest that
LINK is effective for generating data in the long-tail distribution while
enforcing quality. LINT can be useful for systematically evaluating LLMs'
capabilities in the long-tail distribution. We challenge the models with a
simple entailment classification task using samples from LINT. We find that
ChatGPT and GPT4's capability in identifying incorrect knowledge drop by ~3% in
the long-tail distribution compared to head distribution.

大型语言模型，长尾分布，逻辑诱导知识搜索框架，数据集，蕴涵分类。

寻找长尾：通过逻辑规则引导搜索系统生成长尾知识

In Search of the Long-Tail: Systematic Generation of Long-Tail Knowledge  via Logical Rule Guided Search

Textual entailment recognition is one of the basic natural language
understanding(NLU) tasks. Understanding the meaning of sentences is a
prerequisite before applying any natural language processing(NLP) techniques to
automatically recognize the textual entailment. A text entails a hypothesis if
and only if the true value of the hypothesis follows the text. Classical
approaches generally utilize the feature value of each word from word embedding
to represent the sentences. In this paper, we propose a novel approach to
identifying the textual entailment relationship between text and hypothesis,
thereby introducing a new semantic feature focusing on empirical
threshold-based semantic text representation. We employ an element-wise
Manhattan distance vector-based feature that can identify the semantic
entailment relationship between the text-hypothesis pair. We carried out
several experiments on a benchmark entailment classification(SICK-RTE) dataset.
We train several machine learning(ML) algorithms applying both semantic and
lexical features to classify the text-hypothesis pair as entailment, neutral,
or contradiction. Our empirical sentence representation technique enriches the
semantic information of the texts and hypotheses found to be more efficient
than the classical ones. In the end, our approach significantly outperforms
known methods in understanding the meaning of the sentences for the textual
entailment classification task.

本文提出了一种新的基于阈值的语义文本表示方法，并结合机器学习算法应用语义和词汇特征对文本 - 假设对进行分类，该技术有效地丰富了文本的语义信息，在文字蕴涵分类任务中在意义理解方面的表现明显优于已知方法