Continual learning aims to refine model parameters for new tasks while
retaining knowledge from previous tasks. Recently, prompt-based learning has
emerged to leverage pre-trained models to be prompted to learn subsequent tasks
without the reliance on the rehearsal buffer. Although this approach has
demonstrated outstanding results, existing methods depend on preceding
task-selection process to choose appropriate prompts. However, imperfectness in
task-selection may lead to negative impacts on the performance particularly in
the scenarios where the number of tasks is large or task distributions are
imbalanced. To address this issue, we introduce I-Prompt, a task-agnostic
approach focuses on the visual semantic information of image tokens to
eliminate task prediction. Our method consists of semantic prompt matching,
which determines prompts based on similarities between tokens, and image
token-level prompting, which applies prompts directly to image tokens in the
intermediate layers. Consequently, our method achieves competitive performance
on four benchmarks while significantly reducing training time compared to
state-of-the-art methods. Moreover, we demonstrate the superiority of our
method across various scenarios through extensive experiments.

我们介绍了一种名为 I-Prompt 的方法，它是一种与任务无关的方法，专注于图像令牌的视觉语义信息，以消除任务预测，并在四个基准测试中达到了竞争性能，同时与最先进的方法相比显著减少了训练时间。通过大量实验，我们证明了我们的方法在各种场景下的优越性。

通过图像记号进行语义提示的持续学习

Semantic Prompting with Image-Token for Continual Learning

The goal of few-shot relation extraction is to predict relations between name
entities in a sentence when only a few labeled instances are available for
training. Existing few-shot relation extraction methods focus on uni-modal
information such as text only. This reduces performance when there are no clear
contexts between the name entities described in text. We propose a multi-modal
few-shot relation extraction model (MFS-HVE) that leverages both textual and
visual semantic information to learn a multi-modal representation jointly. The
MFS-HVE includes semantic feature extractors and multi-modal fusion components.
The MFS-HVE semantic feature extractors are developed to extract both textual
and visual features. The visual features include global image features and
local object features within the image. The MFS-HVE multi-modal fusion unit
integrates information from various modalities using image-guided attention,
object-guided attention, and hybrid feature attention to fully capture the
semantic interaction between visual regions of images and relevant texts.
Extensive experiments conducted on two public datasets demonstrate that
semantic visual information significantly improves the performance of few-shot
relation prediction.

多模态表示学习在少样本关系抽取上的应用

混合视觉证据下的少样本关系抽取

Few-Shot Relation Extraction with Hybrid Visual Evidence

Text-to-image multimodal tasks, generating/retrieving an image from a given
text description, are extremely challenging tasks since raw text descriptions
cover quite limited information in order to fully describe visually realistic
images. We propose a new visual contextual text representation for
text-to-image multimodal tasks, VICTR, which captures rich visual semantic
information of objects from the text input. First, we use the text description
as initial input and conduct dependency parsing to extract the syntactic
structure and analyse the semantic aspect, including object quantities, to
extract the scene graph. Then, we train the extracted objects, attributes, and
relations in the scene graph and the corresponding geometric relation
information using Graph Convolutional Networks, and it generates text
representation which integrates textual and visual semantic information. The
text representation is aggregated with word-level and sentence-level embedding
to generate both visual contextual word and sentence representation. For the
evaluation, we attached VICTR to the state-of-the-art models in text-to-image
generation.VICTR is easily added to existing models and improves across both
quantitative and qualitative aspects.

本文提出了一种新的视觉上下文文本表示方法，VICTR，用于处理文本到图像的多模态任务，通过使用图卷积网络和文本表征的结合，有效地捕捉了文本语义中的视觉特征信息，实现了在实验中得到的量化和定性的改进。