We investigate the mechanisms of factual recall in the Mamba state space
model. Our work is inspired by previous findings in autoregressive transformer
language models suggesting that their knowledge recall is localized to
particular modules at specific token locations; we therefore ask whether
factual recall in Mamba can be similarly localized. To investigate this, we
conduct four lines of experiments on Mamba. First, we apply causal tracing or
interchange interventions to localize key components inside Mamba that are
responsible for recalling facts, revealing that specific components within
middle layers show strong causal effects at the last token of the subject,
while the causal effect of intervening on later layers is most pronounced at
the last token of the prompt, matching previous findings on autoregressive
transformers. Second, we show that rank-one model editing methods can
successfully insert facts at specific locations, again resembling findings on
transformer models. Third, we examine the linearity of Mamba's representations
of factual relations. Finally we adapt attention-knockout techniques to Mamba
to dissect information flow during factual recall. We compare Mamba directly to
a similar-sized transformer and conclude that despite significant differences
in architectural approach, when it comes to factual recall, the two
architectures share many similarities.

通过四条实验研究了 Mamba 状态空间模型中的实际记忆机制，发现其与自回归变压器语言模型在知识记忆的局部性和特定模块的相关性方面具有相似性，并且通过多种技术验证了 Mamba 的实际关系表示的线性性和信息流动。

定位和编辑 Mamba 中的事实关联

Locating and Editing Factual Associations in Mamba

No previous work has studied the performance of Large Language Models (LLMs)
in the context of Traditional Chinese Medicine (TCM), an essential and distinct
branch of medical knowledge with a rich history. To bridge this gap, we present
a TCM question dataset named TCM-QA, which comprises three question types:
single choice, multiple choice, and true or false, to examine the LLM's
capacity for knowledge recall and comprehensive reasoning within the TCM
domain. In our study, we evaluate two settings of the LLM, zero-shot and
few-shot settings, while concurrently discussing the differences between
English and Chinese prompts. Our results indicate that ChatGPT performs best in
true or false questions, achieving the highest precision of 0.688 while scoring
the lowest precision is 0.241 in multiple-choice questions. Furthermore, we
observed that Chinese prompts outperformed English prompts in our evaluations.
Additionally, we assess the quality of explanations generated by ChatGPT and
their potential contribution to TCM knowledge comprehension. This paper offers
valuable insights into the applicability of LLMs in specialized domains and
paves the way for future research in leveraging these powerful models to
advance TCM.

使用大型语言模型研究中医药学领域的知识召回和综合推理能力，通过 TCM-QA 数据集评估 LLM 在零样本和少样本设置下的表现，讨论中英提示的差异，结果显示 ChatGPT 在判断题中取得最高精度 0.688，多选题中最低精度为 0.241，中文提示在评估中表现更好，同时评估 ChatGPT 生成的解释质量及其对中医药学知识理解的潜在贡献，为 LLM 在专业领域的适用性提供了有价值的见解，并促进了未来基于这些强大模型推进中医药学研究的方向。

探索 ChatGPT 对中医知识的理解

Exploring the Comprehension of ChatGPT in Traditional Chinese Medicine  Knowledge

Recent studies have demonstrated that large language models (LLMs) store
massive factual knowledge within their parameters. But existing LLMs are prone
to hallucinate unintended text due to false or outdated knowledge. Since
retraining LLMs is resource intensive, there has been a growing interest in the
concept of model editing. Despite the emergence of benchmarks and approaches,
these unidirectional editing and evaluation have failed to explore the reversal
curse. Intuitively, if "The capital of France is" is edited to be a counterfact
"London" within a model, then it should be able to naturally reason and recall
the reverse fact, i.e., "London is the capital of" followed by "France" instead
of "England". In this paper, we study bidirectional language model editing,
aiming to provide rigorous model editing evaluation to assess if edited LLMs
can recall the editing knowledge bidirectionally. A new evaluation metric of
reversibility is introduced, and a benchmark dubbed as Bidirectional Assessment
for Knowledge Editing (BAKE) is constructed to evaluate the reversibility of
edited models in recalling knowledge in the reverse direction of editing. We
surprisingly observe that while current editing methods and LLMs can
effectively recall editing facts in the direction of editing, they suffer
serious deficiencies when evaluated in the reverse direction. To mitigate the
reversal curse, a method named Bidirectionally Inversible Relationship moDeling
(BIRD) is proposed. A set of editing objectives that incorporate bidirectional
relationships between subject and object into the updated model weights are
designed. Experiments show that BIRD improves the performance of four
representative LLMs of different sizes via question answering and judgement.

本文研究双向语言模型编辑，旨在提供严格的模型编辑评估，以评估编辑后的语言模型是否能双向回忆编辑的知识。引入了一种新的可逆性评估度量，并构建了一个名为 Bidirectional Assessment for Knowledge Editing (BAKE) 的基准来评估编辑后模型在回忆编辑的知识反向方向上的可逆性。我们惊奇地发现，虽然当前的编辑方法和语言模型在编辑方向上能有效地回忆编辑的事实，但在反向方向上评估时存在严重不足。为了缓解这种逆转的问题，提出了一种名为 Bidirectionally Inversible Relationship moDeling (BIRD) 的方法。设计了一组编辑目标，将主体和客体之间的双向关系纳入更新后的模型权重中。实验证明，BIRD 通过问答和判断提高了四个不同规模的代表性语言模型的性能。