Prompting methods have shown impressive performance in a variety of text
mining tasks and applications, especially few-shot ones. Despite the promising
prospects, the performance of prompting model largely depends on the design of
prompt template and verbalizer. In this work, we propose MetricPrompt, which
eases verbalizer design difficulty by reformulating few-shot text
classification task into text pair relevance estimation task. MetricPrompt
adopts prompting model as the relevance metric, further bridging the gap
between Pre-trained Language Model's (PLM) pre-training objective and text
classification task, making possible PLM's smooth adaption. Taking a training
sample and a query one simultaneously, MetricPrompt captures cross-sample
relevance information for accurate relevance estimation. We conduct experiments
on three widely used text classification datasets across four few-shot
settings. Results show that MetricPrompt outperforms manual verbalizer and
other automatic verbalizer design methods across all few-shot settings,
achieving new state-of-the-art (SOTA) performance.

本研究提出了 MetricPrompt 方法来解决 few-shot 分类任务的 verbalizer 设计难题，将其转化为文本对相关性评估任务，使用 prompting 模型作为相关性度量，取得了超过手动 verbalizer 和其他自动 verbalizer 设计方法的最新 SOTA 表现。

MetricPrompt: 以 Prompting 模型作为 Few-shot 文本分类的相关性评估度量

MetricPrompt: Prompting Model as a Relevance Metric for Few-shot Text  Classification

Saliency maps are a popular approach to creating post-hoc explanations of
image classifier outputs. These methods produce estimates of the relevance of
each pixel to the classification output score, which can be displayed as a
saliency map that highlights important pixels. Despite a proliferation of such
methods, little effort has been made to quantify how good these saliency maps
are at capturing the true relevance of the pixels to the classifier output
(i.e. their "fidelity"). We therefore investigate existing metrics for
evaluating the fidelity of saliency methods (i.e. saliency metrics). We find
that there is little consistency in the literature in how such metrics are
calculated, and show that such inconsistencies can have a significant effect on
the measured fidelity. Further, we apply measures of reliability developed in
the psychometric testing literature to assess the consistency of saliency
metrics when applied to individual saliency maps. Our results show that
saliency metrics can be statistically unreliable and inconsistent, indicating
that comparative rankings between saliency methods generated using such metrics
can be untrustworthy.

本研究探索了现有的用于评估显著图质量（即 “保真度”）的指标，并发现当前文献中这些指标的计算方式缺乏一致性，且可重复性差，说明用这些指标生成的显著图质量排名可能不可信。

显著性指标的合理性检验

Sanity Checks for Saliency Metrics

Ranking models lie at the heart of research on information retrieval (IR).
During the past decades, different techniques have been proposed for
constructing ranking models, from traditional heuristic methods, probabilistic
methods, to modern machine learning methods. Recently, with the advance of deep
learning technology, we have witnessed a growing body of work in applying
shallow or deep neural networks to the ranking problem in IR, referred to as
neural ranking models in this paper. The power of neural ranking models lies in
the ability to learn from the raw text inputs for the ranking problem to avoid
many limitations of hand-crafted features. Neural networks have sufficient
capacity to model complicated tasks, which is needed to handle the complexity
of relevance estimation in ranking. Since there have been a large variety of
neural ranking models proposed, we believe it is the right time to summarize
the current status, learn from existing methodologies, and gain some insights
for future development. In contrast to existing reviews, in this survey, we
will take a deep look into the neural ranking models from different dimensions
to analyze their underlying assumptions, major design principles, and learning
strategies. We compare these models through benchmark tasks to obtain a
comprehensive empirical understanding of the existing techniques. We will also
discuss what is missing in the current literature and what are the promising
and desired future directions.

该研究论文主要讨论了信息检索中的排名模型如何应用传统启发式方法、概率方法和现代机器学习技术等技术构建，并重点介绍了基于浅层或深层神经网络的神经排名模型，分析了其基本原理和学习策略，并通过基准测试比较了这些模型，最后探讨了当前文献中存在的问题以及未来的发展方向。