Aligned text-image encoders such as CLIP have become the de facto model for
vision-language tasks. Furthermore, modality-specific encoders achieve
impressive performances in their respective domains. This raises a central
question: does an alignment exist between uni-modal vision and language
encoders since they fundamentally represent the same physical world? Analyzing
the latent spaces structure of vision and language models on image-caption
benchmarks using the Centered Kernel Alignment (CKA), we find that the
representation spaces of unaligned and aligned encoders are semantically
similar. In the absence of statistical similarity in aligned encoders like
CLIP, we show that a possible matching of unaligned encoders exists without any
training. We frame this as a seeded graph-matching problem exploiting the
semantic similarity between graphs and propose two methods - a Fast Quadratic
Assignment Problem optimization, and a novel localized CKA metric-based
matching/retrieval. We demonstrate the effectiveness of this on several
downstream tasks including cross-lingual, cross-domain caption matching and
image classification.

通过使用 Centered Kernel Alignment (CKA) 分析图像字幕基准上视觉和语言模型的潜在空间结构，我们发现不对齐和对齐的编码器的表示空间在语义上是相似的。在无统计相似性的情况下，我们展示了存在可能匹配不对齐编码器而无需任何训练。我们将其视为一种基于种子图匹配问题，利用图之间的语义相似性提出了两种方法 - 一种是快速二次分配问题优化，一种是基于新颖局部 CKA 度量的匹配 / 检索。我们在包括跨语言、跨域字幕匹配和图像分类在内的几个下游任务上展示了其有效性。

视觉和语言编码器是否相似地代表世界？

Do Vision and Language Encoders Represent the World Similarly?

Compliments and concerns in reviews are valuable for understanding users'
shopping interests and their opinions with respect to specific aspects of
certain items. Existing review-based recommenders favor large and complex
language encoders that can only learn latent and uninterpretable text
representations. They lack explicit user attention and item property modeling,
which however could provide valuable information beyond the ability to
recommend items. Therefore, we propose a tightly coupled two-stage approach,
including an Aspect-Sentiment Pair Extractor (ASPE) and an
Attention-Property-aware Rating Estimator (APRE). Unsupervised ASPE mines
Aspect-Sentiment pairs (AS-pairs) and APRE predicts ratings using AS-pairs as
concrete aspect-level evidence. Extensive experiments on seven real-world
Amazon Review Datasets demonstrate that ASPE can effectively extract AS-pairs
which enable APRE to deliver superior accuracy over the leading baselines.

本文提出了一个紧密耦合的两阶段方法以改进基于评论的推荐系统。通过提取方面 - 情感对和利用它们作为具体的层面证据来估算评级，APRE 方法超越了现有的基线，并有效地提高了精度。

基于无监督的方面 - 情感共同提取：解锁推荐的力量

Recommend for a Reason: Unlocking the Power of Unsupervised  Aspect-Sentiment Co-Extraction

Representational Similarity Analysis (RSA) is a technique developed by
neuroscientists for comparing activity patterns of different measurement
modalities (e.g., fMRI, electrophysiology, behavior). As a framework, RSA has
several advantages over existing approaches to interpretation of language
encoders based on probing or diagnostic classification: namely, it does not
require large training samples, is not prone to overfitting, and it enables a
more transparent comparison between the representational geometries of
different models and modalities. We demonstrate the utility of RSA by
establishing a previously unknown correspondence between widely-employed
pretrained language encoders and human processing difficulty via eye-tracking
data, showcasing its potential in the interpretability toolbox for neural
models

论文介绍了一种称为 RSA 的技术，可以比较不同测量模式下（例如 fMRI、电生理学和行为学）的活动模式，展示了 RSA 的实用性，并通过眼动数据建立了先前未知的预先训练的语言编码器和人类处理难度之间的对应关系，展示了它作为神经模型可解释性工具箱的潜力。