Dynamical systems are often time-varying, whose modeling requires a function
that evolves with respect to time. Recent studies such as the neural ordinary
differential equation proposed a time-dependent neural network, which provides
a neural network varying with respect to time. However, we claim that the
architectural choice to build a time-dependent neural network significantly
affects its time-awareness but still lacks sufficient validation in its current
states. In this study, we conduct an in-depth analysis of the architecture of
modern time-dependent neural networks. Here, we report a vulnerability of
vanishing timestep embedding, which disables the time-awareness of a
time-dependent neural network. Furthermore, we find that this vulnerability can
also be observed in diffusion models because they employ a similar architecture
that incorporates timestep embedding to discriminate between different
timesteps during a diffusion process. Our analysis provides a detailed
description of this phenomenon as well as several solutions to address the root
cause. Through experiments on neural ordinary differential equations and
diffusion models, we observed that ensuring alive time-awareness via proposed
solutions boosted their performance, which implies that their current
implementations lack sufficient time-dependency.

通过深入分析现代时间相关神经网络的结构，研究报告了一个时间相关神经网络的时间感知关键缺陷，该缺陷同样存在于扩散模型中，同时提供了几种解决方案以改善问题根源，并通过对神经常微分方程和扩散模型的实验验证表明，保证时间感知能力可以提升它们的性能，暗示了它们当前的实现缺乏足够的时间依赖性。