Safely navigating street intersections is a complex challenge for blind and
low-vision individuals, as it requires a nuanced understanding of the
surrounding context - a task heavily reliant on visual cues. Traditional
methods for assisting in this decision-making process often fall short, lacking
the ability to provide a comprehensive scene analysis and safety level. This
paper introduces an innovative approach that leverages large multimodal models
(LMMs) to interpret complex street crossing scenes, offering a potential
advancement over conventional traffic signal recognition techniques. By
generating a safety score and scene description in natural language, our method
supports safe decision-making for the blind and low-vision individuals. We
collected crosswalk intersection data that contains multiview egocentric images
captured by a quadruped robot and annotated the images with corresponding
safety scores based on our predefined safety score categorization. Grounded on
the visual knowledge, extracted from images, and text prompt, we evaluate a
large multimodal model for safety score prediction and scene description. Our
findings highlight the reasoning and safety score prediction capabilities of a
LMM, activated by various prompts, as a pathway to developing a trustworthy
system, crucial for applications requiring reliable decision-making support.

通过使用多模式模型解释复杂的街道交叉场景，为盲人和低视力人群提供安全评分和场景描述，支持他们进行安全决策。

可解释风险评估：GPT-4V 在关心安全的过马路中的应用

Is it safe to cross? Interpretable Risk Assessment with GPT-4V for  Safety-Aware Street Crossing

Image reconstruction in low-light conditions is a challenging problem. Many
solutions have been proposed for it, where the main approach is trying to learn
a good prior of natural images along with modeling the true statistics of the
noise in the scene. In the presence of very low lighting conditions, such
approaches are usually not enough, and additional information is required,
e.g., in the form of using multiple captures. In this work, we suggest as an
alternative to add a description of the scene as prior, which can be easily
done by the photographer who is capturing the scene. Using a text-conditioned
diffusion model, we show that adding image caption information improves
significantly the image reconstruction in low-light conditions on both
synthetic and real-world images.

在低光条件下，通过添加场景描述的先验信息，使用文本条件扩散模型，我们在合成和真实世界图像上展示了图像重建在低光条件下的显著改进。

告诉我你所看见的：文本引导的现实世界图像去噪

Tell Me What You See: Text-Guided Real-World Image Denoising

Image generation from scene description is a cornerstone technique for the
controlled generation, which is beneficial to applications such as content
creation and image editing. In this work, we aim to synthesize images from
scene description with retrieved patches as reference. We propose a
differentiable retrieval module. With the differentiable retrieval module, we
can (1) make the entire pipeline end-to-end trainable, enabling the learning of
better feature embedding for retrieval; (2) encourage the selection of mutually
compatible patches with additional objective functions. We conduct extensive
quantitative and qualitative experiments to demonstrate that the proposed
method can generate realistic and diverse images, where the retrieved patches
are reasonable and mutually compatible.

该论文提出一种利用场景描述和参考补丁合成图像的方法，并使用可微分检索模块实现端到端的培训，以学习更好的特征嵌入用于检索，并通过附加目标函数鼓励选择相互兼容的补丁。经过广泛的定量和定性实验表明，该方法可以生成逼真多样的图像，检索的补丁合理且相互兼容。