We propose In-Context Translation (ICT), a general learning framework to
unify visual recognition (e.g., semantic segmentation), low-level image
processing (e.g., denoising), and conditional image generation (e.g.,
edge-to-image synthesis). Thanks to unification, ICT significantly reduces the
inherent inductive bias that comes with designing models for specific tasks,
and it maximizes mutual enhancement across similar tasks. However, the
unification across a large number of tasks is non-trivial due to various data
formats and training pipelines. To this end, ICT introduces two designs.
Firstly, it standardizes input-output data of different tasks into RGB image
pairs, e.g., semantic segmentation data pairs an RGB image with its
segmentation mask in the same RGB format. This turns different tasks into a
general translation task between two RGB images. Secondly, it standardizes the
training of different tasks into a general in-context learning, where
"in-context" means the input comprises an example input-output pair of the
target task and a query image. The learning objective is to generate the
"missing" data paired with the query. The implicit translation process is thus
between the query and the generated image. In experiments, ICT unifies ten
vision tasks and showcases impressive performance on their respective
benchmarks. Notably, compared to its competitors, e.g., Painter and
PromptDiffusion, ICT trained on only 4 RTX 3090 GPUs is shown to be more
efficient and less costly in training.

我们提出了一种称为 In-Context Translation (ICT) 的通用学习框架，用于统一视觉识别、图像处理和条件图像生成。ICT 通过统一减少了特定任务模型设计所带来的内在归纳偏差，并最大程度地增强了相似任务之间的相互促进。在实验中，ICT 统一了十个视觉任务，并在各自的基准测试中展示了令人印象深刻的性能。与其竞争对手（例如 Painter 和 PromptDiffusion）相比，仅使用 4 台 RTX 3090 GPU 对 ICT 进行训练的效率更高且成本更低。