The emergence of large language models (LLMs) has revolutionized numerous
applications across industries. However, their "black box" nature often hinders
the understanding of how they make specific decisions, raising concerns about
their transparency, reliability, and ethical use. This study presents a method
to improve the explainability of LLMs by varying individual words in prompts to
uncover their statistical impact on the model outputs. This approach, inspired
by permutation importance for tabular data, masks each word in the system
prompt and evaluates its effect on the outputs based on the available text
scores aggregated over multiple user inputs. Unlike classical attention, word
importance measures the impact of prompt words on arbitrarily-defined text
scores, which enables decomposing the importance of words into the specific
measures of interest--including bias, reading level, verbosity, etc. This
procedure also enables measuring impact when attention weights are not
available. To test the fidelity of this approach, we explore the effect of
adding different suffixes to multiple different system prompts and comparing
subsequent generations with different large language models. Results show that
word importance scores are closely related to the expected suffix importances
for multiple scoring functions.

通过改变提示中的单词以揭示其对模型输出的统计影响来改善大型语言模型的可解释性，检验结果显示单词重要性得分与多个评分函数的预期后缀重要性密切相关。