It is widely acknowledged that large language models (LLMs) encode a vast
reservoir of knowledge after being trained on mass data. Recent studies
disclose knowledge conflicts in LLM generation, wherein outdated or incorrect
parametric knowledge (i.e., encoded knowledge) contradicts new knowledge
provided in the context. To mitigate such knowledge conflicts, we propose a
novel framework, IRCAN (Identifying and Reweighting Context-Aware Neurons) to
capitalize on neurons that are crucial in processing contextual cues.
Specifically, IRCAN first identifies neurons that significantly contribute to
context processing, utilizing a context-aware attribution score derived from
integrated gradients. Subsequently, the identified context-aware neurons are
strengthened via reweighting. In doing so, we steer LLMs to generate
context-sensitive outputs with respect to the new knowledge provided in the
context. Extensive experiments conducted across a variety of models and tasks
demonstrate that IRCAN not only achieves remarkable improvements in handling
knowledge conflicts but also offers a scalable, plug-andplay solution that can
be integrated seamlessly with existing models.

通过强化关键上下文处理神经元的权重，IRCAN 能够解决大型语言模型在生成文本时面临的知识冲突问题，并且提供可与现有模型无缝集成的扩展解决方案。

IRCAN: 通过识别和重新赋权上下文感知神经元以减少 LLM 生成中的知识冲突

IRCAN: Mitigating Knowledge Conflicts in LLM Generation via Identifying  and Reweighting Context-Aware Neurons

The rapid advancement of Large Language Models (LLMs) has inaugurated a
transformative epoch in natural language processing, fostering unprecedented
proficiency in text generation, comprehension, and contextual scrutiny.
Nevertheless, effectively handling extensive contexts, crucial for myriad
applications, poses a formidable obstacle owing to the intrinsic constraints of
the models' context window sizes and the computational burdens entailed by
their operations. This investigation presents an innovative framework that
strategically tailors LLMs for streamlined context processing by harnessing the
synergies among natural language summarization, soft prompt compression, and
augmented utility preservation mechanisms. Our methodology, dubbed
SoftPromptComp, amalgamates natural language prompts extracted from
summarization methodologies with dynamically generated soft prompts to forge a
concise yet semantically robust depiction of protracted contexts. This
depiction undergoes further refinement via a weighting mechanism optimizing
information retention and utility for subsequent tasks. We substantiate that
our framework markedly diminishes computational overhead and enhances LLMs'
efficacy across various benchmarks, while upholding or even augmenting the
caliber of the produced content. By amalgamating soft prompt compression with
sophisticated summarization, SoftPromptComp confronts the dual challenges of
managing lengthy contexts and ensuring model scalability. Our findings point
towards a propitious trajectory for augmenting LLMs' applicability and
efficiency, rendering them more versatile and pragmatic for real-world
applications. This research enriches the ongoing discourse on optimizing
language models, providing insights into the potency of soft prompts and
summarization techniques as pivotal instruments for the forthcoming generation
of NLP solutions.

该研究提出了一种名为 SoftPromptComp 的创新框架，它通过集成自然语言摘要、软提示压缩和增强型效用保留机制来为大语言模型提供流畅的上下文处理。研究结果表明，该框架显著降低了计算负担，并提高了大语言模型在各项基准测试中的效力，同时保持或增强所生成内容的质量。该研究为优化语言模型提供了见解，并探讨了软提示和摘要技术作为未来自然语言处理解决方案的关键工具的潜力。

通过软提示压缩使 LLMs 适应高效上下文处理

Adapting LLMs for Efficient Context Processing through Soft Prompt  Compression

The widespread diffusion of connected smart devices has contributed to the
rapid expansion and evolution of the Internet at its edge. Personal mobile
devices interact with other smart objects in their surroundings, adapting
behavior based on rapidly changing user context. The ability of mobile devices
to process this data locally is crucial for quick adaptation. This can be
achieved through a single elaboration process integrated into user applications
or a middleware platform for context processing. However, the lack of public
datasets considering user context complexity in the mobile environment hinders
research progress. We introduce MyDigitalFootprint, a large-scale dataset
comprising smartphone sensor data, physical proximity information, and Online
Social Networks interactions. This dataset supports multimodal context
recognition and social relationship modeling. It spans two months of
measurements from 31 volunteer users in their natural environment, allowing for
unrestricted behavior. Existing public datasets focus on limited context data
for specific applications, while ours offers comprehensive information on the
user context in the mobile environment. To demonstrate the dataset's
effectiveness, we present three context-aware applications utilizing various
machine learning tasks: (i) a social link prediction algorithm based on
physical proximity data, (ii) daily-life activity recognition using
smartphone-embedded sensors data, and (iii) a pervasive context-aware
recommender system. Our dataset, with its heterogeneity of information, serves
as a valuable resource to validate new research in mobile and edge computing.

本文介绍了 MyDigitalFootprint 数据集，它包含了智能手机传感器数据、物理接近信息和在线社交网络互动的大规模数据，支持多模态上下文识别和社交关系建模，并通过三个基于机器学习任务的上下文感知应用程序展示了其有效性。