Stakeholders often describe system requirements using natural language which
are then converted to formal syntax by a domain-expert leading to increased
design costs. This paper assesses the capabilities of Large Language Models
(LLMs) in converting between natural language descriptions and formal
specifications. Existing work has evaluated the capabilities of LLMs in
generating formal syntax such as source code but such experiments are typically
hand-crafted and use problems that are likely to be in the training set of
LLMs, and often require human-annotated datasets. We propose an approach that
can use two copies of an LLM in conjunction with an off-the-shelf verifier to
automatically evaluate its translation abilities without any additional human
input. Our approach generates formal syntax using language grammars to
automatically generate a dataset. We conduct an empirical evaluation to measure
the accuracy of this translation task and show that SOTA LLMs cannot adequately
solve this task, limiting their current utility in the design of complex
systems.

利用大型语言模型 (LLMs) 将自然语言描述转换为形式化规范的能力进行了评估，提出了一种使用两个 LLMs 与现成验证程序结合的方法来自动评估其翻译能力，结果显示目前最先进的 LLMs 无法充分解决这个任务，限制了它们在复杂系统设计中的实用性。

LLM 能够正式对话吗？自动评估 LLM 在翻译和解释正式规范中的表现

Can LLMs Converse Formally? Automatically Assessing LLMs in Translating  and Interpreting Formal Specifications

The evolution of autonomous driving has made remarkable advancements in
recent years, evolving into a tangible reality. However, a human-centric
large-scale adoption hinges on meeting a variety of multifaceted requirements.
To ensure that the autonomous system meets the user's intent, it is essential
to accurately discern and interpret user commands, especially in complex or
emergency situations. To this end, we propose to leverage the reasoning
capabilities of Large Language Models (LLMs) to infer system requirements from
in-cabin users' commands. Through a series of experiments that include
different LLM models and prompt designs, we explore the few-shot multivariate
binary classification accuracy of system requirements from natural language
textual commands. We confirm the general ability of LLMs to understand and
reason about prompts but underline that their effectiveness is conditioned on
the quality of both the LLM model and the design of appropriate sequential
prompts. Code and models are public with the link
https://github.com/KTH-RPL/DriveCmd_LLM.

自动驾驶的进化取得了显著的进展，并成为了一个实际存在的现实。为了确保自动驾驶系统符合用户意图，准确辨别和解释用户指令尤其是在复杂或紧急情况下是至关重要的。为了实现这一目标，我们提出利用大型语言模型（LLMs）的推理能力，从车载用户的指令中推断系统需求。通过一系列实验，包括不同的 LLM 模型和提示设计，我们探索了通过自然语言文本指令从少量样本进行多元二值分类的精确度。我们确认 LLM 可以理解和推理提示，但强调其有效性取决于 LLM 模型的质量和适当的连续提示的设计。代码和模型可在以下链接找到：https://github.com/KTH-RPL/DriveCmd_LLM。

以 LLMs 支持用户命令推理的面向人类的自主系统

Human-Centric Autonomous Systems With LLMs for User Command Reasoning

Logical reasoning is essential in a variety of human activities. A
representative example of a logical task is mathematics. Recent large-scale
models trained on large datasets have been successful in various fields, but
their reasoning ability in arithmetic tasks is limited, which we reproduce
experimentally. Here, we recast this limitation as not unique to mathematics
but common to tasks that require logical operations. We then propose a new set
of tasks, termed logical tasks, which will be the next challenge to address.
This higher point of view helps the development of inductive biases that have
broad impact beyond the solution of individual tasks. We define and
characterize logical tasks and discuss system requirements for their solution.
Furthermore, we discuss the relevance of logical tasks to concepts such as
extrapolation, explainability, and inductive bias. Finally, we provide
directions for solving logical tasks.

通过提出逻辑任务的概念，该研究从更高的角度说明了近期大规模模型在算术推理中的局限性，并探讨了解决逻辑任务的方向和需要具备的系统需求。