With the development of multimedia applications, multimodal recommendations
are playing an essential role, as they can leverage rich contexts beyond user
interactions. Existing methods mainly regard multimodal information as an
auxiliary, using them to help learn ID features; however, there exist semantic
gaps among multimodal content features and ID features, for which directly
using multimodal information as an auxiliary would lead to misalignment in
representations of users and items. In this paper, we first systematically
investigate the misalignment issue in multimodal recommendations, and propose a
solution named AlignRec. In AlignRec, the recommendation objective is
decomposed into three alignments, namely alignment within contents, alignment
between content and categorical ID, and alignment between users and items. Each
alignment is characterized by a specific objective function and is integrated
into our multimodal recommendation framework. To effectively train our
AlignRec, we propose starting from pre-training the first alignment to obtain
unified multimodal features and subsequently training the following two
alignments together with these features as input. As it is essential to analyze
whether each multimodal feature helps in training, we design three new classes
of metrics to evaluate intermediate performance. Our extensive experiments on
three real-world datasets consistently verify the superiority of AlignRec
compared to nine baselines. We also find that the multimodal features generated
by AlignRec are better than currently used ones, which are to be open-sourced.

通过系统调查多模态推荐中的对齐问题，本文提出了一种名为 AlignRec 的解决方案，由内容内部对齐、内容与分类 ID 之间的对齐，以及用户与项目之间的对齐这三个目标函数组成，将其融入到我们的多模态推荐框架中，并通过实验证明了 AlignRec 相对于其他九种基线方法的优越性。

多模态推荐的对齐和训练框架

An Aligning and Training Framework for Multimodal Recommendations

The development of large vision-language models (LVLMs) offers the potential
to address challenges faced by traditional multimodal recommendations thanks to
their proficient understanding of static images and textual dynamics. However,
the application of LVLMs in this field is still limited due to the following
complexities: First, LVLMs lack user preference knowledge as they are trained
from vast general datasets. Second, LVLMs suffer setbacks in addressing
multiple image dynamics in scenarios involving discrete, noisy, and redundant
image sequences. To overcome these issues, we propose the novel reasoning
scheme named Rec-GPT4V: Visual-Summary Thought (VST) of leveraging large
vision-language models for multimodal recommendation. We utilize user history
as in-context user preferences to address the first challenge. Next, we prompt
LVLMs to generate item image summaries and utilize image comprehension in
natural language space combined with item titles to query the user preferences
over candidate items. We conduct comprehensive experiments across four datasets
with three LVLMs: GPT4-V, LLaVa-7b, and LLaVa-13b. The numerical results
indicate the efficacy of VST.

利用大型视觉 - 语言模型进行多模态推荐的研究中，研究人员通过引入用户历史和基于图像摘要生成的查询方式，克服了大型视觉 - 语言模型面临的用户偏好知识和多样复杂图像序列问题，并验证了该方法的有效性。