Over decades, neuroscience has accumulated a wealth of research results in
the text modality that can be used to explore cognitive processes.
Meta-analysis is a typical method that successfully establishes a link from
text queries to brain activation maps using these research results, but it
still relies on an ideal query environment. In practical applications, text
queries used for meta-analyses may encounter issues such as semantic redundancy
and ambiguity, resulting in an inaccurate mapping to brain images. On the other
hand, large language models (LLMs) like ChatGPT have shown great potential in
tasks such as context understanding and reasoning, displaying a high degree of
consistency with human natural language. Hence, LLMs could improve the
connection between text modality and neuroscience, resolving existing
challenges of meta-analyses. In this study, we propose a method called
Chat2Brain that combines LLMs to basic text-2-image model, known as Text2Brain,
to map open-ended semantic queries to brain activation maps in data-scarce and
complex query environments. By utilizing the understanding and reasoning
capabilities of LLMs, the performance of the mapping model is optimized by
transferring text queries to semantic queries. We demonstrate that Chat2Brain
can synthesize anatomically plausible neural activation patterns for more
complex tasks of text queries.

该研究提出了一种名为 Chat2Brain 的方法，将大型语言模型与基本的文本 - 图像模型 Text2Brain 相结合，以在数据稀缺和复杂查询环境中将开放式语义查询映射到脑激活图，从而改善元分析的挑战。我们证明 Chat2Brain 能够合成更复杂的文本查询任务的解剖学上合理的神经激活模式。

Chat2Brain: 将开放性语义查询映射为脑活动图的方法

Chat2Brain: A Method for Mapping Open-Ended Semantic Queries to Brain  Activation Maps

We address the problem of retrieving images with both a sketch and a text
query. We present TASK-former (Text And SKetch transformer), an end-to-end
trainable model for image retrieval using a text description and a sketch as
input. We argue that both input modalities complement each other in a manner
that cannot be achieved easily by either one alone. TASK-former follows the
late-fusion dual-encoder approach, similar to CLIP, which allows efficient and
scalable retrieval since the retrieval set can be indexed independently of the
queries. We empirically demonstrate that using an input sketch (even a poorly
drawn one) in addition to text considerably increases retrieval recall compared
to traditional text-based image retrieval. To evaluate our approach, we collect
5,000 hand-drawn sketches for images in the test set of the COCO dataset. The
collected sketches are available a this https URL

本文介绍了一个基于文本及草图的图像检索模型，该模型称为 “TASK-former”，并采用了类似于 CLIP 的融合编码器的方法，众多实验表明，使用草图可以明显提高图像检索的召回率。

草图胜千言：带文本和草图的图像检索

A Sketch Is Worth a Thousand Words: Image Retrieval with Text and Sketch

Associating image regions with text queries has been recently explored as a
new way to bridge visual and linguistic representations. A few pioneering
approaches have been proposed based on recurrent neural language models trained
generatively (e.g., generating captions), but achieving somewhat limited
localization accuracy. To better address natural-language-based visual entity
localization, we propose a discriminative approach. We formulate a
discriminative bimodal neural network (DBNet), which can be trained by a
classifier with extensive use of negative samples. Our training objective
encourages better localization on single images, incorporates text phrases in a
broad range, and properly pairs image regions with text phrases into positive
and negative examples. Experiments on the Visual Genome dataset demonstrate the
proposed DBNet significantly outperforms previous state-of-the-art methods both
for localization on single images and for detection on multiple images. We we
also establish an evaluation protocol for natural-language visual detection.

该研究提出了一种基于区分式双模神经网络 (DBNet) 的方法来实现自然语言及视觉表述中的视觉实体定位，通过为分类器提供丰富的负例样本进行训练，结合广泛的文本短语，正确地将图像区域与文本短语成对为正负例，据实验证明，该方法比现有最先进方法在单个图像和多个图像上的定位以及检测都要更为准确。