Policy documents, such as legislation, regulations, and executive orders, are
crucial in shaping society. However, their length and complexity make
interpretation and application challenging and time-consuming. Artificial
intelligence (AI), particularly large language models (LLMs), has the potential
to automate the process of analyzing these documents, improving accuracy and
efficiency. This study aims to evaluate the potential of AI in streamlining
policy analysis and to identify the strengths and limitations of current AI
approaches. The research focuses on question answering and tasks involving
content extraction from policy documents. A case study was conducted using
Executive Order 14110 on "Safe, Secure, and Trustworthy Development and Use of
Artificial Intelligence" as a test case. Four commercial AI systems were used
to analyze the document and answer a set of representative policy questions.
The performance of the AI systems was compared to manual analysis conducted by
human experts. The study found that two AI systems, Gemini 1.5 Pro and Claude 3
Opus, demonstrated significant potential for supporting policy analysis,
providing accurate and reliable information extraction from complex documents.
They performed comparably to human analysts but with significantly higher
efficiency. However, achieving reproducibility remains a challenge,
necessitating further research and development.

研究评估了人工智能在简化政策分析中的潜力，并识别了当前人工智能方法的优点和限制。研究还通过使用《关于 “人工智能的安全、可靠和可信开发与应用”》这一行政令进行案例研究，对四个商业人工智能系统进行了性能比较。研究发现，Gemini 1.5 Pro 和 Claude 3 Opus 两个人工智能系统在支持政策分析方面表现出了显著的潜力，能够从复杂文件中提取准确可靠的信息，其效率与人类分析师相当，但明显更高。然而，实现可复现性仍然存在挑战，需要进一步的研究和开发。

利用人工智能高效分析复杂政策文件：一份《行政命令 14110》案例研究

Harnessing AI for efficient analysis of complex policy documents: a case  study of Executive Order 14110

Classifying policy documents into policy issue topics has been a long-time
effort in political science and communication disciplines. Efforts to automate
text classification processes for social science research purposes have so far
achieved remarkable results, but there is still a large room for progress. In
this work, we test the prediction performance of an alternative strategy, which
requires human involvement much less than full manual coding. We use the GPT
3.5 and GPT 4 models of the OpenAI, which are pre-trained instruction-tuned
Large Language Models (LLM), to classify congressional bills and congressional
hearings into Comparative Agendas Project's 21 major policy issue topics. We
propose three use-case scenarios and estimate overall accuracies ranging from
%58-83 depending on scenario and GPT model employed. The three scenarios aims
at minimal, moderate, and major human interference, respectively. Overall, our
results point towards the insufficiency of complete reliance on GPT with
minimal human intervention, an increasing accuracy along with the human effort
exerted, and a surprisingly high accuracy achieved in the most humanly
demanding use-case. However, the superior use-case achieved the %83 accuracy on
the %65 of the data in which the two models agreed, suggesting that a similar
approach to ours can be relatively easily implemented and allow for mostly
automated coding of a majority of a given dataset. This could free up resources
allowing manual human coding of the remaining %35 of the data to achieve an
overall higher level of accuracy while reducing costs significantly.

使用 GPT 3.5 和 GPT 4 模型对议会法案和听证会进行分类，与人工干预程度相关的三种使用情景中，我们的结果表明完全依赖最小化人工干预的 GPT 不够充分，人工投入程度越高准确性越高，并在最需要人工干预的情况下取得了令人惊讶的高准确率。然而，优秀的使用情景中的 83% 准确率仅在两个模型达成一致的 65% 数据上实现，暗示着类似我们的方法相对容易实施，并可实现对大多数给定数据集的自动编码，从而节省资源并降低成本。