Event cameras are advantageous for tasks that require vision sensors with
low-latency and sparse output responses. However, the development of deep
network algorithms using event cameras has been slow because of the lack of
large labelled event camera datasets for network training. This paper reports a
method for creating new labelled event datasets by using a text-to-X model,
where X is one or multiple output modalities, in the case of this work, events.
Our proposed text-to-events model produces synthetic event frames directly from
text prompts. It uses an autoencoder which is trained to produce sparse event
frames representing event camera outputs. By combining the pretrained
autoencoder with a diffusion model architecture, the new text-to-events model
is able to generate smooth synthetic event streams of moving objects. The
autoencoder was first trained on an event camera dataset of diverse scenes. In
the combined training with the diffusion model, the DVS gesture dataset was
used. We demonstrate that the model can generate realistic event sequences of
human gestures prompted by different text statements. The classification
accuracy of the generated sequences, using a classifier trained on the real
dataset, ranges between 42% to 92%, depending on the gesture group. The results
demonstrate the capability of this method in synthesizing event datasets.

使用文本生成模型和扩散模型，本研究提出了一种用于创建新的标记事件数据集的方法，通过合成事件帧直接从文本提示生成平滑的移动对象事件流。根据不同文本语句，该模型能够生成真实人体动作的逼真事件序列，其分类准确率在 42% 至 92% 之间，展示出合成事件数据集的能力。

文本转事件：有条件文本输入的合成事件相机流

Text-to-Events: Synthetic Event Camera Streams from Conditional Text  Input

Event datasets are sequences of events of various types occurring irregularly
over the time-line, and they are increasingly prevalent in numerous domains.
Existing work for modeling events using conditional intensities rely on either
using some underlying parametric form to capture historical dependencies, or on
non-parametric models that focus primarily on tasks such as prediction. We
propose a non-parametric deep neural network approach in order to estimate the
underlying intensity functions. We use a novel multi-channel RNN that optimally
reinforces the negative evidence of no observable events with the introduction
of fake event epochs within each consecutive inter-event interval. We evaluate
our method against state-of-the-art baselines on model fitting tasks as gauged
by log-likelihood. Through experiments on both synthetic and real-world
datasets, we find that our proposed approach outperforms existing baselines on
most of the datasets studied.

该论文提出了一种基于多通道 RNN 的非参数深度神经网络方法，用于对事件数据集进行建模和预测，并证明该方法在模型拟合任务中胜过现有的基线方法。