The development of large vision-language models (LVLMs) offers the potential
to address challenges faced by traditional multimodal recommendations thanks to
their proficient understanding of static images and textual dynamics. However,
the application of LVLMs in this field is still limited due to the following
complexities: First, LVLMs lack user preference knowledge as they are trained
from vast general datasets. Second, LVLMs suffer setbacks in addressing
multiple image dynamics in scenarios involving discrete, noisy, and redundant
image sequences. To overcome these issues, we propose the novel reasoning
scheme named Rec-GPT4V: Visual-Summary Thought (VST) of leveraging large
vision-language models for multimodal recommendation. We utilize user history
as in-context user preferences to address the first challenge. Next, we prompt
LVLMs to generate item image summaries and utilize image comprehension in
natural language space combined with item titles to query the user preferences
over candidate items. We conduct comprehensive experiments across four datasets
with three LVLMs: GPT4-V, LLaVa-7b, and LLaVa-13b. The numerical results
indicate the efficacy of VST.

利用大型视觉 - 语言模型进行多模态推荐的研究中，研究人员通过引入用户历史和基于图像摘要生成的查询方式，克服了大型视觉 - 语言模型面临的用户偏好知识和多样复杂图像序列问题，并验证了该方法的有效性。

Rec-GPT4V：大型视觉语言模型的多模态推荐

Rec-GPT4V: Multimodal Recommendation with Large Vision-Language Models

Multimodal recommendation aims to model user and item representations
comprehensively with the involvement of multimedia content for effective
recommendations. Existing research has shown that it is beneficial for
recommendation performance to combine (user- and item-) ID embeddings with
multimodal salient features, indicating the value of IDs. However, there is a
lack of a thorough analysis of the ID embeddings in terms of feature semantics
in the literature. In this paper, we revisit the value of ID embeddings for
multimodal recommendation and conduct a thorough study regarding its semantics,
which we recognize as subtle features of content and structures. Then, we
propose a novel recommendation model by incorporating ID embeddings to enhance
the semantic features of both content and structures. Specifically, we put
forward a hierarchical attention mechanism to incorporate ID embeddings in
modality fusing, coupled with contrastive learning, to enhance content
representations. Meanwhile, we propose a lightweight graph convolutional
network for each modality to amalgamate neighborhood and ID embeddings for
improving structural representations. Finally, the content and structure
representations are combined to form the ultimate item embedding for
recommendation. Extensive experiments on three real-world datasets (Baby,
Sports, and Clothing) demonstrate the superiority of our method over
state-of-the-art multimodal recommendation methods and the effectiveness of
fine-grained ID embeddings.

本文研究了多模态推荐中 ID 嵌入的价值和语义特征，并提出了一种新的推荐模型，通过引入 ID 嵌入来增强内容和结构的语义特征。通过层次化注意机制和对比学习，提高了内容表示，并利用轻量级图卷积网络和 ID 嵌入改进了结构表示，最后将内容和结构表示相结合形成最终的物品嵌入。实验结果证明了我们方法在多个真实数据集上的优越性和对精细 ID 嵌入的有效性。

多模态推荐的内容和结构的微妙特征中的 ID 嵌入

ID Embedding as Subtle Features of Content and Structure for Multimodal  Recommendation

Multimedia content is of predominance in the modern Web era. Investigating
how users interact with multimodal items is a continuing concern within the
rapid development of recommender systems. The majority of previous work focuses
on modeling user-item interactions with multimodal features included as side
information. However, this scheme is not well-designed for multimedia
recommendation. Specifically, only collaborative item-item relationships are
implicitly modeled through high-order item-user-item relations. Considering
that items are associated with rich contents in multiple modalities, we argue
that the latent semantic item-item structures underlying these multimodal
contents could be beneficial for learning better item representations and
further boosting recommendation. To this end, we propose a LATent sTructure
mining method for multImodal reCommEndation, which we term LATTICE for brevity.
To be specific, in the proposed LATTICE model, we devise a novel modality-aware
structure learning layer, which learns item-item structures for each modality
and aggregates multiple modalities to obtain latent item graphs. Based on the
learned latent graphs, we perform graph convolutions to explicitly inject
high-order item affinities into item representations. These enriched item
representations can then be plugged into existing collaborative filtering
methods to make more accurate recommendations. Extensive experiments on three
real-world datasets demonstrate the superiority of our method over
state-of-the-art multimedia recommendation methods and validate the efficacy of
mining latent item-item relationships from multimodal features.

本研究提出了一种基于 LATent sTructure mining method for multImodal reCommEndation（即 LATTICE）的方法，该方法从多种模式中挖掘潜在语义项目 - 项目结构，以学习更好的项目表示形式并进一步推动多媒体推荐，实验表明该方法可以提高推荐准确性。