As Large Language Models (LLMs) are nondeterministic, the same input can
generate different outputs, some of which may be incorrect or hallucinated. If
run again, the LLM may correct itself and produce the correct answer.
Unfortunately, most LLM-powered systems resort to single results which, correct
or not, users accept. Having the LLM produce multiple outputs may help identify
disagreements or alternatives. However, it is not obvious how the user will
interpret conflicts or inconsistencies. To this end, we investigate how users
perceive the AI model and comprehend the generated information when they
receive multiple, potentially inconsistent, outputs. Through a preliminary
study, we identified five types of output inconsistencies. Based on these
categories, we conducted a study (N=252) in which participants were given one
or more LLM-generated passages to an information-seeking question. We found
that inconsistency within multiple LLM-generated outputs lowered the
participants' perceived AI capacity, while also increasing their comprehension
of the given information. Specifically, we observed that this positive effect
of inconsistencies was most significant for participants who read two passages,
compared to those who read three. Based on these findings, we present design
implications that, instead of regarding LLM output inconsistencies as a
drawback, we can reveal the potential inconsistencies to transparently indicate
the limitations of these models and promote critical LLM usage.

通过研究测试人员对于多个、潜在不一致的模型生成输出的理解，我们发现不一致降低了被测试人员对 AI 的感知能力，同时也提高了对给定信息的理解。因此，我们提出了设计建议，将不一致性透明地呈现出来，以指示这些模型的局限性，并促进批判性的语言模型使用。