Automatic image colorization is inherently an ill-posed problem with
uncertainty, which requires an accurate semantic understanding of scenes to
estimate reasonable colors for grayscale images. Although recent
interaction-based methods have achieved impressive performance, it is still a
very difficult task to infer realistic and accurate colors for automatic
colorization. To reduce the difficulty of semantic understanding of grayscale
scenes, this paper tries to utilize corresponding audio, which naturally
contains extra semantic information about the same scene. Specifically, a novel
audio-infused automatic image colorization (AIAIC) network is proposed, which
consists of three stages. First, we take color image semantics as a bridge and
pretrain a colorization network guided by color image semantics. Second, the
natural co-occurrence of audio and video is utilized to learn the color
semantic correlations between audio and visual scenes. Third, the implicit
audio semantic representation is fed into the pretrained network to finally
realize the audio-guided colorization. The whole process is trained in a
self-supervised manner without human annotation. In addition, an audiovisual
colorization dataset is established for training and testing. Experiments
demonstrate that audio guidance can effectively improve the performance of
automatic colorization, especially for some scenes that are difficult to
understand only from visual modality.

通过利用音频的额外语义信息，本文提出了一种新颖的自动图像上色网络（AIAIC），它由三个阶段组成，通过颜色图像语义作为桥梁的预训练、学习音频和视觉场景之间的颜色语义相关性以及利用隐式音频语义表示实现音频引导的上色，实验表明音频引导能够有效改善自动着色的性能，特别是对于那些仅仅从视觉模态理解起来比较困难的场景。

通过利用音频场景语义实现自动图像着色

Audio-Infused Automatic Image Colorization by Exploiting Audio Scene  Semantics

Deep neural networks for automatic image colorization often suffer from the
color-bleeding artifact, a problematic color spreading near the boundaries
between adjacent objects. Such color-bleeding artifacts debase the reality of
generated outputs, limiting the applicability of colorization models in
practice. Although previous approaches have attempted to address this problem
in an automatic manner, they tend to work only in limited cases where a high
contrast of gray-scale values are given in an input image. Alternatively,
leveraging user interactions would be a promising approach for solving this
color-breeding artifacts. In this paper, we propose a novel edge-enhancing
network for the regions of interest via simple user scribbles indicating where
to enhance. In addition, our method requires a minimal amount of effort from
users for their satisfactory enhancement. Experimental results demonstrate that
our interactive edge-enhancing approach effectively improves the color-bleeding
artifacts compared to the existing baselines across various datasets.

本文提出了一种基于用户交互的边缘增强网络，通过简单的用户涂鸦指示增强哪些感兴趣区域，实现对色彩渗透问题的有效改善，且只需要用户进行最小的努力。实验结果表明，与现有基线相比，我们的交互式边缘增强方法在各种数据集上有效地改善了色彩渗透问题。