Holistic understanding and reasoning in 3D scenes play a vital role in the
success of autonomous driving systems. The evolution of 3D semantic occupancy
prediction as a pretraining task for autonomous driving and robotic downstream
tasks captures finer 3D details compared to methods like 3D detection. Existing
approaches predominantly focus on spatial cues, often overlooking temporal
cues. Query-based methods tend to converge on computationally intensive Voxel
representation for encoding 3D scene information. This study introduces
S2TPVFormer, an extension of TPVFormer, utilizing a spatiotemporal transformer
architecture for coherent 3D semantic occupancy prediction. Emphasizing the
importance of spatiotemporal cues in 3D scene perception, particularly in 3D
semantic occupancy prediction, our work explores the less-explored realm of
temporal cues. Leveraging Tri-Perspective View (TPV) representation, our
spatiotemporal encoder generates temporally rich embeddings, improving
prediction coherence while maintaining computational efficiency. To achieve
this, we propose a novel Temporal Cross-View Hybrid Attention (TCVHA)
mechanism, facilitating effective spatiotemporal information exchange across
TPV views. Experimental evaluations on the nuScenes dataset demonstrate a
substantial 3.1% improvement in mean Intersection over Union (mIoU) for 3D
Semantic Occupancy compared to TPVFormer, confirming the effectiveness of the
proposed S2TPVFormer in enhancing 3D scene perception.

本研究针对自主驾驶系统中对三维场景的整体理解和推理的重要性展开研究，将三维语义占据预测作为自主驾驶和机器人下游任务的预训练任务，与三维检测等方法相比，它更准确地捕捉了三维细节。现有方法主要关注空间线索，而忽视了时间线索。基于查询的方法常常使用计算密集的体素表示对三维场景信息进行编码。本研究介绍了 S2TPVFormer，它是 TPVFormer 的扩展，利用时空变换器结构进行一致的三维语义占据预测。强调了时空线索在三维场景感知中的重要性，特别是在三维语义占据预测中，我们的工作探索了较少研究的时间线索领域。通过利用三视角视图（Tri-Perspective View，简称 TPV）表示，我们的时空编码器生成具有时间上下文的嵌入，提高了预测的一致性，同时保持了计算效率。为了实现这一点，我们提出了一种新颖的时空跨视图混合注意机制（Temporal Cross-View Hybrid Attention，简称 TCVHA），促进了 TPV 视图之间的有效时空信息交流。对 nuScenes 数据集的实验评估表明，与 TPVFormer 相比，提出的 S2TPVFormer 在三维语义占据上取得了 3.1% 的平均交集联合（mIoU）改进，验证了其提高三维场景感知效果的有效性。

S2TPVFormer：时空三视角视图用于具有时间一致性的 3D 语义占据预测

S2TPVFormer: Spatio-Temporal Tri-Perspective View for temporally  coherent 3D Semantic Occupancy Prediction

While the abuse of deepfake technology has caused serious concerns recently,
how to detect deepfake videos is still a challenge due to the high
photo-realistic synthesis of each frame. Existing image-level approaches often
focus on single frame and ignore the spatiotemporal cues hidden in deepfake
videos, resulting in poor generalization and robustness. The key of a
video-level detector is to fully exploit the spatiotemporal inconsistency
distributed in local facial regions across different frames in deepfake videos.
Inspired by that, this paper proposes a simple yet effective patch-level
approach to facilitate deepfake video detection via spatiotemporal dropout
transformer. The approach reorganizes each input video into bag of patches that
is then fed into a vision transformer to achieve robust representation.
Specifically, a spatiotemporal dropout operation is proposed to fully explore
patch-level spatiotemporal cues and serve as effective data augmentation to
further enhance model's robustness and generalization ability. The operation is
flexible and can be easily plugged into existing vision transformers. Extensive
experiments demonstrate the effectiveness of our approach against 25
state-of-the-arts with impressive robustness, generalizability, and
representation ability.

本文提出了一种基于空时 dropout 的深度学习算法用于视频检测，利用视觉变换器从视频中抽取局部区域的空时信息，配合空时 dropout 操作以提高模型的鲁棒性和泛化性。实验证明，该算法有效地解决了当前深度伪造视频的检测挑战。