Artificial intelligence has demonstrated immense potential in scientific
research. Within molecular science, it is revolutionizing the traditional
computer-aided paradigm, ushering in a new era of deep learning. With recent
progress in multimodal learning and natural language processing, an emerging
trend has targeted at building multimodal frameworks to jointly model molecules
with textual domain knowledge. In this paper, we present the first systematic
survey on multimodal frameworks for molecules research. Specifically,we begin
with the development of molecular deep learning and point out the necessity to
involve textual modality. Next, we focus on recent advances in text-molecule
alignment methods, categorizing current models into two groups based on their
architectures and listing relevant pre-training tasks. Furthermore, we delves
into the utilization of large language models and prompting techniques for
molecular tasks and present significant applications in drug discovery.
Finally, we discuss the limitations in this field and highlight several
promising directions for future research.

通过综述多模态分子研究的框架，本文探讨了人工智能在分子科学中的应用，特别关注文本与分子之间的对齐方法以及使用大型语言模型和提示技术进行分子任务的重要应用，同时讨论了该领域的局限性和未来研究的几个有希望的方向。

文本与分子之间的桥梁：一项关于分子多模态框架的调查

Bridging Text and Molecule: A Survey on Multimodal Frameworks for  Molecule

With the increasing availability of depth sensors, multimodal frameworks that
combine color information with depth data are attracting increasing interest.
In the challenging task of semantic segmentation, depth maps allow to
distinguish between similarly colored objects at different depths and provide
useful geometric cues. On the other side, ground truth data for semantic
segmentation is burdensome to be provided and thus domain adaptation is another
significant research area. Specifically, we address the challenging source-free
domain adaptation setting where the adaptation is performed without reusing
source data. We propose MISFIT: MultImodal Source-Free Information fusion
Transformer, a depth-aware framework which injects depth information into a
segmentation module based on vision transformers at multiple stages, namely at
the input, feature and output levels. Color and depth style transfer helps
early-stage domain alignment while re-wiring self-attention between modalities
creates mixed features allowing the extraction of better semantic content.
Furthermore, a depth-based entropy minimization strategy is also proposed to
adaptively weight regions at different distances. Our framework, which is also
the first approach using vision transformers for source-free semantic
segmentation, shows noticeable performance improvements with respect to
standard strategies.

研究基于深度传感器的多模态框架和基于视觉变换器的源自由语义分割的深度感知方法，通过混合特征和深度信息以及深度基础熵最小化方法，能够提高分割效果。

无源域适应方法在 RGB-D 语义分割中的应用：基于 Vision Transformers

Source-Free Domain Adaptation for RGB-D Semantic Segmentation with  Vision Transformers

Vision-language models can assess visual context in an image and generate
descriptive text. While the generated text may be accurate and syntactically
correct, it is often overly general. To address this, recent work has used
optical character recognition to supplement visual information with text
extracted from an image. In this work, we contend that vision-language models
can benefit from additional information that can be extracted from an image,
but are not used by current models. We modify previous multimodal frameworks to
accept relevant information from any number of auxiliary classifiers. In
particular, we focus on person names as an additional set of tokens and create
a novel image-caption dataset to facilitate captioning with person names. The
dataset, Politicians and Athletes in Captions (PAC), consists of captioned
images of well-known people in context. By fine-tuning pretrained models with
this dataset, we demonstrate a model that can naturally integrate facial
recognition tokens into generated text by training on limited data. For the PAC
dataset, we provide a discussion on collection and baseline benchmark scores.

本文主要介绍了一种将人名加入生成文本的新方法，通过使用 OCR 识别图像中的文字并 fine-tuning 预训练模型，我们的方法在生成文本的时候自然地加入了人名信息。为了达到这个目的，我们修改了之前的多模态框架，接受来自任意数量的辅助分类器提供的相关信息。同时，我们创建了一个新的图像 - 标题数据集，名为 PAC，这个数据集包含了一些知名人物的图像和对这些图像的描述，这些描述中包含了人名。