Product image segmentation is vital in e-commerce. Most existing methods
extract the product image foreground only based on the visual modality, making
it difficult to distinguish irrelevant products. As product titles contain
abundant appearance information and provide complementary cues for product
image segmentation, we propose a mutual query network to segment products based
on both visual and linguistic modalities. First, we design a language query
vision module to obtain the response of language description in image areas,
thus aligning the visual and linguistic representations across modalities.
Then, a vision query language module utilizes the correlation between visual
and linguistic modalities to filter the product title and effectively suppress
the content irrelevant to the vision in the title. To promote the research in
this field, we also construct a Multi-Modal Product Segmentation dataset
(MMPS), which contains 30,000 images and corresponding titles. The proposed
method significantly outperforms the state-of-the-art methods on MMPS.

本文提出一种基于视觉和语言两种形式的模态的互相查询网络，用于产品图像的分割，并构建了一个多模态产品分割数据集（MMPS），实验结果表明该方法显著优于现有方法。

多模态产品图像分割的互查询网络

Mutual Query Network for Multi-Modal Product Image Segmentation

This paper investigates the effectiveness and implementation of
modality-specific large-scale pre-trained encoders for multimodal sentiment
analysis~(MSA). Although the effectiveness of pre-trained encoders in various
fields has been reported, conventional MSA methods employ them for only
linguistic modality, and their application has not been investigated. This
paper compares the features yielded by large-scale pre-trained encoders with
conventional heuristic features. One each of the largest pre-trained encoders
publicly available for each modality are used; CLIP-ViT, WavLM, and BERT for
visual, acoustic, and linguistic modalities, respectively. Experiments on two
datasets reveal that methods with domain-specific pre-trained encoders attain
better performance than those with conventional features in both unimodal and
multimodal scenarios. We also find it better to use the outputs of the
intermediate layers of the encoders than those of the output layer. The codes
are available at this https URL.

本文研究了多模态情感分析中使用具有模态感受的大规模预训练编码器的有效性和实现。通过实验比较了在视觉、声学和语言感受模态下最大的预训练编码器和传统的启发式特征后，得出领域专用预训练编码器比常规特征在单模态和多模态情况下性能更好的结论。

使用模态专用的大规模预训练编码器进行多模态情感分析

On the Use of Modality-Specific Large-Scale Pre-Trained Encoders for Multimodal Sentiment Analysis

Understanding linguistic modality is widely seen as important for downstream
tasks such as Question Answering and Knowledge Graph Population. Entailment
Graph learning might also be expected to benefit from attention to modality. We
build Entailment Graphs using a news corpus filtered with a modality parser,
and show that stripping modal modifiers from predicates in fact increases
performance. This suggests that for some tasks, the pragmatics of modal
modification of predicates allows them to contribute as evidence of entailment.

通过使用情态解析器过滤新闻语料库建立蕴涵图，实验证明去除谓词中的情态修饰符（linguistic modality）可提高性能，表明在某些任务中，谓词的情态修饰在实现蕴涵证据方面是有用的（pragmatics of modal modification of predicates）。