Graphs with abundant attributes are essential in modeling interconnected
entities and improving predictions in various real-world applications.
Traditional Graph Neural Networks (GNNs), which are commonly used for modeling
attributed graphs, need to be re-trained every time when applied to different
graph tasks and datasets. Although the emergence of Large Language Models
(LLMs) has introduced a new paradigm in natural language processing, the
generative potential of LLMs in graph mining remains largely under-explored. To
this end, we propose a novel framework MuseGraph, which seamlessly integrates
the strengths of GNNs and LLMs and facilitates a more effective and generic
approach for graph mining across different tasks and datasets. Specifically, we
first introduce a compact graph description via the proposed adaptive input
generation to encapsulate key information from the graph under the constraints
of language token limitations. Then, we propose a diverse instruction
generation mechanism, which distills the reasoning capabilities from LLMs
(e.g., GPT-4) to create task-specific Chain-of-Thought-based instruction
packages for different graph tasks. Finally, we propose a graph-aware
instruction tuning with a dynamic instruction package allocation strategy
across tasks and datasets, ensuring the effectiveness and generalization of the
training process. Our experimental results demonstrate significant improvements
in different graph tasks, showcasing the potential of our MuseGraph in
enhancing the accuracy of graph-oriented downstream tasks while keeping the
generation powers of LLMs.

提出了一种新颖的框架 MuseGraph，它无缝地整合了 GNN 和 LLM 的优势，通过自适应输入生成和多样化的指令生成机制，以及针对不同任务和数据集的图感知指令调优，提高了图任务的准确性，并保持了 LLM 的生成能力。

MuseGraph: 面向通用图挖掘的大型语言模型的图导向指令调优

MuseGraph: Graph-oriented Instruction Tuning of Large Language Models  for Generic Graph Mining

"When did the emperor Napoleon invented iPhone?" Such hallucination-inducing
question is well known challenge in generative language modeling. In this
study, we present an innovative concept of visual hallucination, referred to as
"I Know (IK)" hallucination, to address scenarios where "I Don't Know" is the
desired response. To effectively tackle this issue, we propose the VQAv2-IDK
benchmark, the subset of VQAv2 comprising unanswerable image-question pairs as
determined by human annotators. Stepping further, we present the visually
dehallucinative instruction generation method for IK hallucination and
introduce the IDK-Instructions visual instruction database. Our experiments
show that current methods struggle with IK hallucination. Yet, our approach
effectively reduces these hallucinations, proving its versatility across
different frameworks and datasets.

在这项研究中，我们提出了一种创新的视觉幻觉概念，称为 “我知道（IK）” 幻觉，以解决所需回答为 “我不知道” 的场景。为了有效应对这个问题，我们提出了 VQAv2-IDK 基准，这是由人类标注人员确定的不可回答的图像问题对的子集。更进一步，我们提出了用于 IK 幻觉的视觉解幻指令生成方法，并介绍了 IDK-Instructions 视觉指令数据库。我们的实验证明了当前方法在处理 IK 幻觉时存在困难，然而我们的方法有效减少了这些幻觉，证明了它在不同框架和数据集上的多功能性。

视觉去想象指导生成：了解自己不知道的知识

Visually Dehallucinative Instruction Generation: Know What You Don't  Know

Since the rise of vision-language navigation (VLN), great progress has been
made in instruction following -- building a follower to navigate environments
under the guidance of instructions. However, far less attention has been paid
to the inverse task: instruction generation -- learning a speaker~to generate
grounded descriptions for navigation routes. Existing VLN methods train a
speaker independently and often treat it as a data augmentation tool to
strengthen the follower while ignoring rich cross-task relations. Here we
describe an approach that learns the two tasks simultaneously and exploits
their intrinsic correlations to boost the training of each: the follower judges
whether the speaker-created instruction explains the original navigation route
correctly, and vice versa. Without the need of aligned instruction-path pairs,
such cycle-consistent learning scheme is complementary to task-specific
training targets defined on labeled data, and can also be applied over
unlabeled paths (sampled without paired instructions). Another agent,
called~creator is added to generate counterfactual environments. It greatly
changes current scenes yet leaves novel items -- which are vital for the
execution of original instructions -- unchanged. Thus more informative training
scenes are synthesized and the three agents compose a powerful VLN learning
system. Extensive experiments on a standard benchmark show that our approach
improves the performance of various follower models and produces accurate
navigation instructions.

本篇研究提出了一种基于视觉 - 语言导航（VLN）的学习方法，同时学习指令生成和指令跟随任务并通过相互的协作来提高训练，包括添加一个能够生成反事实环境来提高训练效果的代理程序。在标准测试集上的实验证明，该方法提高了各种跟随模型的性能并能够生成准确的导航指令。