Commanding a drone with a natural language is not only user-friendly but also
opens the door for emerging language agents to control the drone. Emerging
large language models (LLMs) provide a previously impossible opportunity to
automatically translate a task description in a natural language to a program
that can be executed by the drone. However, powerful LLMs and their vision
counterparts are limited in three important ways. First, they are only
available as cloud-based services. Sending images to the cloud raises privacy
concerns. Second, they are expensive, costing proportionally to the request
size. Finally, without expensive fine-tuning, existing LLMs are quite limited
in their capability of writing a program for specialized systems like drones.
In this paper, we present a system called TypeFly that tackles the above
three problems using a combination of edge-based vision intelligence, novel
programming language design, and prompt engineering. Instead of the familiar
Python, TypeFly gets a cloud-based LLM service to write a program in a small,
custom language called MiniSpec, based on task and scene descriptions in
English. Such MiniSpec programs are not only succinct (and therefore efficient)
but also able to consult the LLM during their execution using a special skill
called query. Using a set of increasingly challenging drone tasks, we show that
design choices made by TypeFly can reduce both the cost of LLM service and the
task execution time by more than 2x. More importantly, query and prompt
engineering techniques contributed by TypeFly significantly improve the chance
of success of complex tasks.

用自然语言操控无人机不仅用户友好，还为新兴的语言代理控制无人机打开了大门。使用基于边缘视觉智能、新颖的编程语言设计和提示工程的系统 TypeFly，通过组合任务和场景描述，TypeFly 利用云端大型语言模型服务以一种小型自定义语言 MiniSpec 编写程序，在执行过程中还能利用查询技巧与语言模型进行交互，从而显著降低了 LLM 服务成本和任务执行时间，并大大提高了复杂任务的成功率。