Architectural Knowledge Management (AKM) involves the organized handling of
information related to architectural decisions and design within a project or
organization. An essential artifact of AKM is the Architecture Decision Records
(ADR), which documents key design decisions. ADRs are documents that capture
decision context, decision made and various aspects related to a design
decision, thereby promoting transparency, collaboration, and understanding.
Despite their benefits, ADR adoption in software development has been slow due
to challenges like time constraints and inconsistent uptake. Recent
advancements in Large Language Models (LLMs) may help bridge this adoption gap
by facilitating ADR generation. However, the effectiveness of LLM for ADR
generation or understanding is something that has not been explored. To this
end, in this work, we perform an exploratory study that aims to investigate the
feasibility of using LLM for the generation of ADRs given the decision context.
In our exploratory study, we utilize GPT and T5-based models with 0-shot,
few-shot, and fine-tuning approaches to generate the Decision of an ADR given
its Context. Our results indicate that in a 0-shot setting, state-of-the-art
models such as GPT-4 generate relevant and accurate Design Decisions, although
they fall short of human-level performance. Additionally, we observe that more
cost-effective models like GPT-3.5 can achieve similar outcomes in a few-shot
setting, and smaller models such as Flan-T5 can yield comparable results after
fine-tuning. To conclude, this exploratory study suggests that LLM can generate
Design Decisions, but further research is required to attain human-level
generation and establish standardized widespread adoption.

通过使用大型语言模型，我们进行了一项探索性研究，旨在研究在给定决策背景的情况下，使用 LLM 生成 ADR 的可行性。我们的研究结果表明，在零 - shot 设定中，GPT-4 等先进模型可以生成相关且准确的设计决策，但它们尚未达到人类水平。此外，我们观察到，成本更低的模型如 GPT-3.5 在一些 - shot 设定中可以获得类似的结果，并且经过精调的更小模型如 Flan-T5 在生成上具有可比性。总之，这项探索性研究表明 LLM 可以生成设计决策，但需要进一步的研究以达到人类水平的生成，并确立标准化的广泛应用。