A multimodal AI agent is characterized by its ability to process and learn
from various types of data, including natural language, visual, and audio
inputs, to inform its actions. Despite advancements in large language models
that incorporate visual data, such as GPT-4V, effectively translating
image-based data into actionable outcomes for AI agents continues to be
challenging. In this paper, we introduce a multimodal model that incorporates
the concept of functional token specifically designed for AI agent
applications. To ensure compatibility with edge devices, our model is optimized
to a compact size of less than 1B parameters. Like GPT-4, our model can process
both English and Chinese. We demonstrate that this model is capable of
operating efficiently on a wide range of edge devices, including as constrained
as a Raspberry Pi.

多模态 AI 代理通过处理和学习来自自然语言、视觉和音频输入的各种数据来指导行动，但将基于图像的数据转换为可操作结果仍然具有挑战性。本文介绍了一种多模态模型，其中包含专门为 AI 代理应用设计的功能令牌的概念，并优化为小于 1B 参数的紧凑尺寸，能够在各种边缘设备上高效运行，包括树莓派。