In human conversations, individuals can indicate relevant regions within a
scene while addressing others. In turn, the other person can then respond by
referring to specific regions if necessary. This natural referential ability in
dialogue remains absent in current Multimodal Large Language Models (MLLMs). To
fill this gap, this paper proposes an MLLM called Shikra, which can handle
spatial coordinate inputs and outputs in natural language. Its architecture
consists of a vision encoder, an alignment layer, and a LLM. It is designed to
be straightforward and simple, without the need for extra vocabularies,
position encoder, pre-/post-detection modules, or external plug-in models. All
inputs and outputs are in natural language form. Referential dialogue is a
superset of various vision-language (VL) tasks. Shikra can naturally handle
location-related tasks like REC and PointQA, as well as conventional VL tasks
such as Image Captioning and VQA. Experimental results showcase Shikra's
promising performance. Furthermore, it enables numerous exciting applications,
like providing mentioned objects' coordinates in chains of thoughts and
comparing user-pointed regions similarities. Our code and model are accessed at
this https URL

本文介绍一种名为 Shikra 的 MLLM 模型，具备自然语言处理输入输出并能处理视觉相关任务，包括定位相关的任务，如 REC 和 PointQA，以及常规的视觉语言任务，如图像字幕和 VQA，实验结果表明其性能良好，能够进行指定物体的坐标提供，并比较用户指定区域的相似性。

Shikra: 发挥多模 LLM 的指代对话魔力

Shikra: Unleashing Multimodal LLM's Referential Dialogue Magic

Biomedical entity linking aims to map biomedical mentions, such as diseases
and drugs, to standard entities in a given knowledge base. The specific
challenge in this context is that the same biomedical entity can have a wide
range of names, including synonyms, morphological variations, and names with
different word orderings. Recently, BERT-based methods have advanced the
state-of-the-art by allowing for rich representations of word sequences.
However, they often have hundreds of millions of parameters and require heavy
computing resources, which limits their applications in resource-limited
scenarios. Here, we propose a lightweight neural method for biomedical entity
linking, which needs just a fraction of the parameters of a BERT model and much
less computing resources. Our method uses a simple alignment layer with
attention mechanisms to capture the variations between mention and entity
names. Yet, we show that our model is competitive with previous work on
standard evaluation benchmarks.

本研究提出了一种轻量级神经方法，利用对齐层和注意力机制来捕捉提到与实体的不同名称变化，以解决生物医学概念链接中同一实体具有各种名称变体的挑战，并且在标准评估基准上证明该模型表现竞争力。