Knowledge editing for large language models can offer an efficient solution
to alter a model's behavior without negatively impacting the overall
performance. However, the current approach encounters issues with limited
generalizability across tasks, necessitating one distinct editor for each task,
which significantly hinders the broader applications. To address this, we take
the first step to analyze the multi-task generalization issue in knowledge
editing. Specifically, we develop an instruction-based editing technique,
termed InstructEdit, which facilitates the editor's adaptation to various task
performances simultaneously using simple instructions. With only one unified
editor for each LLM, we empirically demonstrate that InstructEdit can improve
the editor's control, leading to an average 14.86% increase in Reliability in
multi-task editing setting. Furthermore, experiments involving holdout unseen
task illustrate that InstructEdit consistently surpass previous strong
baselines. To further investigate the underlying mechanisms of
instruction-based knowledge editing, we analyze the principal components of the
editing gradient directions, which unveils that instructions can help control
optimization direction with stronger OOD generalization. Code and datasets will
be available in this https URL

大规模语言模型的知识编辑可以提供一种有效的解决方案，以改变模型的行为而不会对整体性能产生负面影响。为了解决当前方法在任务上的有限泛化能力的问题，我们采取了第一步来分析知识编辑中的多任务泛化问题。具体而言，我们开发了一种基于指令的编辑技术，称为 InstructEdit，它通过简单的指令同时促进编辑器对不同任务性能的适应。通过仅使用一个统一的编辑器为每个大规模语言模型，我们经验证明 InstructEdit 可以提高编辑器的控制能力，在多任务编辑设置中可平均提高 14.86％的可靠性。此外，涉及未见任务的实验表明 InstructEdit 始终优于先前的强基准。为了进一步研究基于指令的知识编辑的基本机制，我们分析了编辑梯度方向的主要成分，发现指令可以帮助控制具有更强的无法覆盖的泛化能力的优化方向。代码和数据集将在此 https URL 中提供。

InstructEdit：用于大型语言模型的指令基础知识编辑

InstructEdit: Instruction-based Knowledge Editing for Large Language  Models

In this paper, we present ChatPLUG, a Chinese open-domain dialogue system for
digital human applications that instruction finetunes on a wide range of
dialogue tasks in a unified internet-augmented format. Different from other
open-domain dialogue models that focus on large-scale pre-training and scaling
up model size or dialogue corpus, we aim to build a powerful and practical
dialogue system for digital human with diverse skills and good multi-task
generalization by internet-augmented instruction tuning. To this end, we first
conduct large-scale pre-training on both common document corpus and dialogue
data with curriculum learning, so as to inject various world knowledge and
dialogue abilities into ChatPLUG. Then, we collect a wide range of dialogue
tasks spanning diverse features of knowledge, personality, multi-turn memory,
and empathy, on which we further instruction tune \modelname via unified
natural language instruction templates. External knowledge from an internet
search is also used during instruction finetuning for alleviating the problem
of knowledge hallucinations. We show that \modelname outperforms
state-of-the-art Chinese dialogue systems on both automatic and human
evaluation, and demonstrates strong multi-task generalization on a variety of
text understanding and generation tasks. In addition, we deploy \modelname to
real-world applications such as Smart Speaker and Instant Message applications
with fast inference. Our models and code will be made publicly available on
ModelScope~\footnote{\small{this https URL}}
and Github~\footnote{\small{this https URL}}.

本文介绍了 ChatPLUG：一个用于数字人类应用的中文开放域对话系统，通过互联网增强的指导微调在统一的互联网增强格式中针对广泛的对话任务。在多个任务上进行指导微调，集成外部知识的互联网搜索，从而表现出良好的多任务泛化性能，并在真实世界应用中展示出卓越的性能。

ChatPLUG: 面向数字人类的开放领域生成对话系统，利用互联网增强的指令调整

ChatPLUG: Open-Domain Generative Dialogue System with Internet-Augmented  Instruction Tuning for Digital Human

Despite the success of recent Neural Architecture Search (NAS) methods on
various tasks which have shown to output networks that largely outperform
human-designed networks, conventional NAS methods have mostly tackled the
optimization of searching for the network architecture for a single task
(dataset), which does not generalize well across multiple tasks (datasets).
Moreover, since such task-specific methods search for a neural architecture
from scratch for every given task, they incur a large computational cost, which
is problematic when the time and monetary budget are limited. In this paper, we
propose an efficient NAS framework that is trained once on a database
consisting of datasets and pretrained networks and can rapidly search for a
neural architecture for a novel dataset. The proposed MetaD2A (Meta
Dataset-to-Architecture) model can stochastically generate graphs
(architectures) from a given set (dataset) via a cross-modal latent space
learned with amortized meta-learning. Moreover, we also propose a
meta-performance predictor to estimate and select the best architecture without
direct training on target datasets. The experimental results demonstrate that
our model meta-learned on subsets of ImageNet-1K and architectures from
NAS-Bench 201 search space successfully generalizes to multiple unseen datasets
including CIFAR-10 and CIFAR-100, with an average search time of 33 GPU
seconds. Even under MobileNetV3 search space, MetaD2A is 5.5K times faster than
NSGANetV2, a transferable NAS method, with comparable performance. We believe
that the MetaD2A proposes a new research direction for rapid NAS as well as
ways to utilize the knowledge from rich databases of datasets and architectures
accumulated over the past years. Code is available at
this https URL

本文提出了一种高效的神经架构搜索（Neural Architecture Search, NAS）框架，基于元学习的跨模态潜空间，结合预训练的网络库和数据库，可以快速地为新数据集搜索神经架构，并且还提出了一种元性能预测器来从多个未见数据集中预估和选择最佳的神经架构。实验结果表明，该模型受过 ImageNet-1K 的元学习并结合 NAS-Bench 201 搜索空间的神经架构，在包括 CIFAR-10 和 CIFAR-100 等多个未见数据集上成功泛化，平均搜索时间为 33 GPU 秒，即使在 MobileNetV3 搜索空间下，它比具有相似性能的可转移 NAS 方法 NSGANetV2 快 5.5K 倍。我们认为这为快速 NAS 和利用累积多年的数据集和架构的知识提出了新的研究方向。