Scene depth estimation from paintings can streamline the process of 3D
sculpture creation so that visually impaired people appreciate the paintings
with tactile sense. However, measuring depth of oriental landscape painting
images is extremely challenging due to its unique method of depicting depth and
poor preservation. To address the problem of scene depth estimation from
oriental landscape painting images, we propose a novel framework that consists
of two-step Image-to-Image translation method with CLIP-based image matching at
the front end to predict the real scene image that best matches with the given
oriental landscape painting image. Then, we employ a pre-trained SOTA depth
estimation model for the generated real scene image. In the first step,
CycleGAN converts an oriental landscape painting image into a pseudo-real scene
image. We utilize CLIP to semantically match landscape photo images with an
oriental landscape painting image for training CycleGAN in an unsupervised
manner. Then, the pseudo-real scene image and oriental landscape painting image
are fed into DiffuseIT to predict a final real scene image in the second step.
Finally, we measure depth of the generated real scene image using a pre-trained
depth estimation model such as MiDaS. Experimental results show that our
approach performs well enough to predict real scene images corresponding to
oriental landscape painting images. To the best of our knowledge, this is the
first study to measure the depth of oriental landscape painting images. Our
research potentially assists visually impaired people in experiencing paintings
in diverse ways. We will release our code and resulting dataset.

通过 Image-to-Image 翻译方法和基于 CLIP 的图像匹配，我们提出了一个新的框架来估计东方山水画图像的场景深度，最终生成与画作对应的真实场景图像，为视觉障碍者提供多样的欣赏方式。

传统东方风景画的场景深度估计

Scene Depth Estimation from Traditional Oriental Landscape Paintings

We present a method that learns a spatiotemporal neural irradiance field for
dynamic scenes from a single video. Our learned representation enables
free-viewpoint rendering of the input video. Our method builds upon recent
advances in implicit representations. Learning a spatiotemporal irradiance
field from a single video poses significant challenges because the video
contains only one observation of the scene at any point in time. The 3D
geometry of a scene can be legitimately represented in numerous ways since
varying geometry (motion) can be explained with varying appearance and vice
versa. We address this ambiguity by constraining the time-varying geometry of
our dynamic scene representation using the scene depth estimated from video
depth estimation methods, aggregating contents from individual frames into a
single global representation. We provide an extensive quantitative evaluation
and demonstrate compelling free-viewpoint rendering results.

该研究提出了一种从单个视频学习动态场景的时空神经辐射场的方法，能实现自由视角渲染输入视频。该学习方法建立在最近关于内隐表示的进展之上，利用视频深度估计方法的场景深度来约束我们的动态场景表示的时变几何，将单个帧中内容聚合为一个全局表示，并提供了广泛的定量评估和引人注目的自由视点渲染结果。