The explosion of e-commerce has caused the need for processing and analysis
of product titles, like entity typing in product titles. However, the rapid
activity in e-commerce has led to the rapid emergence of new entities, which is
difficult to be solved by general entity typing. Besides, product titles in
e-commerce have very different language styles from text data in general
domain. In order to handle new entities in product titles and address the
special language styles problem of product titles in e-commerce domain, we
propose our textual entailment model with continuous prompt tuning based
hypotheses and fusion embeddings for e-commerce entity typing. First, we
reformulate the entity typing task into a textual entailment problem to handle
new entities that are not present during training. Second, we design a model to
automatically generate textual entailment hypotheses using a continuous prompt
tuning method, which can generate better textual entailment hypotheses without
manual design. Third, we utilize the fusion embeddings of BERT embedding and
CharacterBERT embedding with a two-layer MLP classifier to solve the problem
that the language styles of product titles in e-commerce are different from
that of general domain. To analyze the effect of each contribution, we compare
the performance of entity typing and textual entailment model, and conduct
ablation studies on continuous prompt tuning and fusion embeddings. We also
evaluate the impact of different prompt template initialization for the
continuous prompt tuning. We show our proposed model improves the average F1
score by around 2% compared to the baseline BERT entity typing model.

为应对电子商务领域商品标题语言特点与新实体的问题，我们提出了基于连续提示调整、融合嵌入的文本蕴含模型来进行电商中实体类型标注，模型改进了 BERT 模型的平均 F1 得分约 2%。

基于 Continuous Prompt Tuning 的文本蕴含模型在电子商务实体类型识别中的应用

Continuous Prompt Tuning Based Textual Entailment Model for E-commerce Entity Typing

Prefix-tuning, or more generally continuous prompt tuning, has become an
essential paradigm of parameter-efficient transfer learning. Using a large
pre-trained language model (PLM), prefix-tuning can obtain strong performance
by training only a small portion of parameters. In this paper, we propose to
understand and further develop prefix-tuning through the kernel lens.
Specifically, we make an analogy between \textit{prefixes} and \textit{inducing
variables} in kernel methods and hypothesize that \textit{prefixes} serving as
\textit{inducing variables} would improve their overall mechanism. From the
kernel estimator perspective, we suggest a new variant of prefix-tuning --
\textit{inducer-tuning}, which shares the exact mechanism as prefix-tuning
while leveraging the residual form found in adapter-tuning. This mitigates the
initialization issue in prefix-tuning. Through comprehensive empirical
experiments on natural language understanding and generation tasks, we
demonstrate that inducer-tuning can close the performance gap between
prefix-tuning and fine-tuning.

本文提出了通过核方法理解 prefix-tuning，并将其发展为 inducer-tuning 来提高其性能，通过自然语言理解和生成任务的综合实验，证明了 inducer-tuning 可以缩小 prefix-tuning 和 fine-tuning 之间的性能差距。

Inducer-tuning: 连接 Prefix-tuning 和 Adapter-tuning

Inducer-tuning: Connecting Prefix-tuning and Adapter-tuning

Parameter-efficient tuning aims to distill knowledge for downstream tasks by
optimizing a few introduced parameters while freezing the pretrained language
models (PLMs). Continuous prompt tuning which prepends a few trainable vectors
to the embeddings of input is one of these methods and has drawn much attention
due to its effectiveness and efficiency. This family of methods can be
illustrated as exerting nonlinear transformations of hidden states inside PLMs.
However, a natural question is ignored: can the hidden states be directly used
for classification without changing them? In this paper, we aim to answer this
question by proposing a simple tuning method which only introduces three
trainable vectors. Firstly, we integrate all layers hidden states using the
introduced vectors. And then, we input the integrated hidden state(s) to a
task-specific linear classifier to predict categories. This scheme is similar
to the way ELMo utilises hidden states except that they feed the hidden states
to LSTM-based models. Although our proposed tuning scheme is simple, it
achieves comparable performance with prompt tuning methods like P-tuning and
P-tuning v2, verifying that original hidden states do contain useful
information for classification tasks. Moreover, our method has an advantage
over prompt tuning in terms of time and the number of parameters.

本篇研究提出一种简单的优化方法，只引入三个可训练向量，并将所有层的隐藏状态使用这些向量进行整合，之后将整合后的隐藏状态输入到一个任务特定的线性分类器中进行分类，通过验证原始的隐藏状态确实包含分类任务的有用信息，证明了该优化方案与 P-tuning v2 相比具有时间和参数数量的优势。