Video Internet of Things (VIoT) has shown full potential in collecting an
unprecedented volume of video data. Learning to schedule perceiving models and
analyzing the collected videos intelligently will be potential sparks for VIoT.
In this paper, to address the challenges posed by the fine-grained and
interrelated vision tool usage of VIoT, we build VIoTGPT, the framework based
on LLMs to correctly interact with humans, query knowledge videos, and invoke
vision models to accomplish complicated tasks. To support VIoTGPT and related
future works, we meticulously crafted the training dataset and established
benchmarks involving 11 representative vision models across three categories
based on semi-automatic annotations. To guide LLM to act as the intelligent
agent towards intelligent VIoT, we resort to ReAct instruction tuning based on
the collected VIoT dataset to learn the tool capability. Quantitative and
qualitative experimental results and analyses demonstrate the effectiveness of
VIoTGPT.

基于 LLMs 的框架 VIoTGPT 可以与人类正确互动、查询知识视频并调用视觉模型完成复杂任务，通过 ReAct 指令调优学习工具能力，定量和定性实验结果验证了 VIoTGPT 的有效性。