Humans show an innate capability to identify tools to support specific
actions. The association between objects parts and the actions they facilitate
is usually named affordance. Being able to segment objects parts depending on
the tasks they afford is crucial to enable intelligent robots to use objects of
daily living. Traditional supervised learning methods for affordance
segmentation require costly pixel-level annotations, while weakly supervised
approaches, though less demanding, still rely on object-interaction examples
and support a closed set of actions. These limitations hinder scalability, may
introduce biases, and usually restrict models to a limited set of predefined
actions. This paper proposes AffordanceCLIP, to overcome these limitations by
leveraging the implicit affordance knowledge embedded within large pre-trained
Vision-Language models like CLIP. We experimentally demonstrate that CLIP,
although not explicitly trained for affordances detection, retains valuable
information for the task. Our AffordanceCLIP achieves competitive zero-shot
performance compared to methods with specialized training, while offering
several advantages: i) it works with any action prompt, not just a predefined
set; ii) it requires training only a small number of additional parameters
compared to existing solutions and iii) eliminates the need for direct
supervision on action-object pairs, opening new perspectives for
functionality-based reasoning of models.

通过利用在预训练的视觉 - 语言模型中内嵌的隐式适应性知识，本文提出了 AffordanceCLIP 方法，克服了传统监督学习方法和弱监督方法在适应性分割方面的限制，并实现了与专门训练方法相竞争的零样本性能，同时为模型的功能性推理打开了新的视角。

CLIP 对剥香蕉的了解

What does CLIP know about peeling a banana?

Structured chemical reaction information plays a vital role for chemists
engaged in laboratory work and advanced endeavors such as computer-aided drug
design. Despite the importance of extracting structured reactions from
scientific literature, data annotation for this purpose is cost-prohibitive due
to the significant labor required from domain experts. Consequently, the
scarcity of sufficient training data poses an obstacle to the progress of
related models in this domain. In this paper, we propose ReactIE, which
combines two weakly supervised approaches for pre-training. Our method utilizes
frequent patterns within the text as linguistic cues to identify specific
characteristics of chemical reactions. Additionally, we adopt synthetic data
from patent records as distant supervision to incorporate domain knowledge into
the model. Experiments demonstrate that ReactIE achieves substantial
improvements and outperforms all existing baselines.

本篇研究文章提出了一种名为 ReactIE 的方法，通过弱监督预训练的方式，结合来自文本中的频繁模式和专利记录中的数据，以识别化学反应的特定特征。实验证明，ReactIE 优于所有现有的基准模型。

ReactIE：利用弱监督增强化学反应提取

ReactIE: Enhancing Chemical Reaction Extraction with Weak Supervision

Few-shot semantic segmentation aims to learn to segment unseen class objects
with the guidance of only a few support images. Most previous methods rely on
the pixel-level label of support images. In this paper, we focus on a more
challenging setting, in which only the image-level labels are available. We
propose a general framework to firstly generate coarse masks with the help of
the powerful vision-language model CLIP, and then iteratively and mutually
refine the mask predictions of support and query images. Extensive experiments
on PASCAL-5i and COCO-20i datasets demonstrate that our method not only
outperforms the state-of-the-art weakly supervised approaches by a significant
margin, but also achieves comparable or better results to recent supervised
methods. Moreover, our method owns an excellent generalization ability for the
images in the wild and uncommon classes. Code will be available at
this https URL

本文提出了一种利用视觉 - 语言模型 CLIP 生成粗略掩模并迭代互相调整支持和查询图片的掩模预测的框架，实验结果表明该方法不仅在 PASCAL-5i 和 COCO-20i 数据集上优于最先进的弱监督方法，还能够取得与最近的有监督方法相当甚至更好的结果，并且具有出色的野外图像和非常规类别的泛化能力。