Vision Transformers (ViTs) have become increasingly popular in large-scale
Vision and Language Pre-training (VLP) models. Although previous VLP research
has demonstrated the efficacy of ViTs, these efforts still struggle with
computational inefficiencies caused by lengthy visual sequences. To address
this challenge, we introduce an efficient VLP approach called TRIPS, which
stands for Text-Relevant Image Patch Selection. TRIPS progressively reduces the
visual sequence using a text-guided patch-selection layer in the visual
backbone, thereby accelerating both training and inference processes. This
patch-selection layer dynamically computes text-dependent visual attention,
enabling it to identify attentive image tokens with text guidance and fuse
inattentive ones in an end-to-end fashion. Importantly, TRIPS does not add any
extra parameters and generalizes to most ViT-based VLP models. We incorporate
TRIPS into three representative VLP models covering single-stream, dual-stream,
and generative paradigms, and conduct extensive experiments on five widely-used
multi-modal benchmark datasets. Our experimental results reveal that TRIPS
delivers a 40% speedup, while maintaining competitive or superior performance
on downstream tasks.

TRIPS 是一种高效的 VLP 方法，通过在视觉主干中引入一个文本导向的补丁选择层，以渐进的方式减少视觉序列，从而加速训练和推理过程，并通过动态计算文本相关的视觉注意力来识别注意力图像令牌，以无缝融合不注意的令牌。TRIPS 不添加额外参数且适用于大多数基于 ViT 的 VLP 模型。将 TRIPS 整合到三种代表性的 VLP 模型中，并在五个广泛使用的多模式基准数据集上进行了详尽的实验研究。实验结果表明，TRIPS 能够提供 40% 的加速效果，同时在下游任务上保持有竞争力或更好的性能。

基于文本相关图像修补选择的高效视觉与语言预训练

Efficient Vision-and-Language Pre-training with Text-Relevant Image  Patch Selection

Point-based radiance field rendering has demonstrated impressive results for
novel view synthesis, offering a compelling blend of rendering quality and
computational efficiency. However, also latest approaches in this domain are
not without their shortcomings. 3D Gaussian Splatting [Kerbl and Kopanas et al.
2023] struggles when tasked with rendering highly detailed scenes, due to
blurring and cloudy artifacts. On the other hand, ADOP [R\"uckert et al. 2022]
can accommodate crisper images, but the neural reconstruction network decreases
performance, it grapples with temporal instability and it is unable to
effectively address large gaps in the point cloud.
In this paper, we present TRIPS (Trilinear Point Splatting), an approach that
combines ideas from both Gaussian Splatting and ADOP. The fundamental concept
behind our novel technique involves rasterizing points into a screen-space
image pyramid, with the selection of the pyramid layer determined by the
projected point size. This approach allows rendering arbitrarily large points
using a single trilinear write. A lightweight neural network is then used to
reconstruct a hole-free image including detail beyond splat resolution.
Importantly, our render pipeline is entirely differentiable, allowing for
automatic optimization of both point sizes and positions.
Our evaluation demonstrate that TRIPS surpasses existing state-of-the-art
methods in terms of rendering quality while maintaining a real-time frame rate
of 60 frames per second on readily available hardware. This performance extends
to challenging scenarios, such as scenes featuring intricate geometry,
expansive landscapes, and auto-exposed footage.

本文介绍了一种名为 TRIPS 的新技术，将点云渲染技术与轻量级神经网络相结合，实现了更高的渲染质量和实时帧率，并适用于复杂场景和自动曝光片段。