Event cameras are advantageous for tasks that require vision sensors with
low-latency and sparse output responses. However, the development of deep
network algorithms using event cameras has been slow because of the lack of
large labelled event camera datasets for network training. This paper reports a
method for creating new labelled event datasets by using a text-to-X model,
where X is one or multiple output modalities, in the case of this work, events.
Our proposed text-to-events model produces synthetic event frames directly from
text prompts. It uses an autoencoder which is trained to produce sparse event
frames representing event camera outputs. By combining the pretrained
autoencoder with a diffusion model architecture, the new text-to-events model
is able to generate smooth synthetic event streams of moving objects. The
autoencoder was first trained on an event camera dataset of diverse scenes. In
the combined training with the diffusion model, the DVS gesture dataset was
used. We demonstrate that the model can generate realistic event sequences of
human gestures prompted by different text statements. The classification
accuracy of the generated sequences, using a classifier trained on the real
dataset, ranges between 42% to 92%, depending on the gesture group. The results
demonstrate the capability of this method in synthesizing event datasets.

使用文本生成模型和扩散模型，本研究提出了一种用于创建新的标记事件数据集的方法，通过合成事件帧直接从文本提示生成平滑的移动对象事件流。根据不同文本语句，该模型能够生成真实人体动作的逼真事件序列，其分类准确率在 42% 至 92% 之间，展示出合成事件数据集的能力。