Grounded Multimodal Named Entity Recognition (GMNER) task aims to identify
named entities, entity types and their corresponding visual regions. GMNER task
exhibits two challenging attributes: 1) The tenuous correlation between images
and text on social media contributes to a notable proportion of named entities
being ungroundable. 2) There exists a distinction between coarse-grained noun
phrases used in similar tasks (e.g., phrase localization) and fine-grained
named entities. In this paper, we propose RiVEG, a unified framework that
reformulates GMNER into a joint MNER-VE-VG task by leveraging large language
models (LLMs) as connecting bridges. This reformulation brings two benefits: 1)
It enables us to optimize the MNER module for optimal MNER performance and
eliminates the need to pre-extract region features using object detection
methods, thus naturally addressing the two major limitations of existing GMNER
methods. 2) The introduction of Entity Expansion Expression module and Visual
Entailment (VE) module unifies Visual Grounding (VG) and Entity Grounding (EG).
This endows the proposed framework with unlimited data and model scalability.
Furthermore, to address the potential ambiguity stemming from the
coarse-grained bounding box output in GMNER, we further construct the new
Segmented Multimodal Named Entity Recognition (SMNER) task and corresponding
Twitter-SMNER dataset aimed at generating fine-grained segmentation masks, and
experimentally demonstrate the feasibility and effectiveness of using box
prompt-based Segment Anything Model (SAM) to empower any GMNER model with the
ability to accomplish the SMNER task. Extensive experiments demonstrate that
RiVEG significantly outperforms SoTA methods on four datasets across the MNER,
GMNER, and SMNER tasks.

提出了 RiVEG，这是一个统一的框架，将 GMNER 转化为联合的 MNER-VE-VG 任务，通过利用大型语言模型作为连接桥梁，解决了 GMNER 任务中存在的挑战，并通过构建新的 SMNER 任务和 Twitter-SMNER 数据集，实验证明了框架的可行性和有效性。

基于 LLM 的改进和基于框的分割的地面多模态命名实体识别的推进

Advancing Grounded Multimodal Named Entity Recognition via LLM-Based  Reformulation and Box-Based Segmentation

Grounded Multimodal Named Entity Recognition (GMNER) is a nascent multimodal
task that aims to identify named entities, entity types and their corresponding
visual regions. GMNER task exhibits two challenging properties: 1) The weak
correlation between image-text pairs in social media results in a significant
portion of named entities being ungroundable. 2) There exists a distinction
between coarse-grained referring expressions commonly used in similar tasks
(e.g., phrase localization, referring expression comprehension) and
fine-grained named entities. In this paper, we propose RiVEG, a unified
framework that reformulates GMNER into a joint MNER-VE-VG task by leveraging
large language models (LLMs) as a connecting bridge. This reformulation brings
two benefits: 1) It maintains the optimal MNER performance and eliminates the
need for employing object detection methods to pre-extract regional features,
thereby naturally addressing two major limitations of existing GMNER methods.
2) The introduction of entity expansion expression and Visual Entailment (VE)
Module unifies Visual Grounding (VG) and Entity Grounding (EG). It enables
RiVEG to effortlessly inherit the Visual Entailment and Visual Grounding
capabilities of any current or prospective multimodal pretraining models.
Extensive experiments demonstrate that RiVEG outperforms state-of-the-art
methods on the existing GMNER dataset and achieves absolute leads of 10.65%,
6.21%, and 8.83% in all three subtasks.

RiVEG 是一个统一的框架，通过利用大型语言模型作为连接桥梁，将 GMNER 重新构建为一个联合的 MNER-VE-VG 任务。该框架能够通过引入实体扩展表达和视觉蕴涵模块，自然地解决现有 GMNER 方法的两个主要限制，并在现有 GMNER 数据集上优于现有方法，三个子任务中分别达到 10.65%，6.21%，和 8.83% 的绝对领先优势。