We propose Unified-IO, a model that performs a large variety of AI tasks
spanning classical computer vision tasks, including pose estimation, object
detection, depth estimation and image generation, vision-and-language tasks
such as region captioning and referring expression, to natural language
processing tasks such as question answering and paraphrasing. Developing a
single unified model for such a large variety of tasks poses unique challenges
due to the heterogeneous inputs and outputs pertaining to each task, including
RGB images, per-pixel maps, binary masks, bounding boxes, and language. We
achieve this unification by homogenizing every supported input and output into
a sequence of discrete vocabulary tokens. This common representation across all
tasks allows us to train a single transformer-based architecture, jointly on
over 90 diverse datasets in the vision and language fields. Unified-IO is the
first model capable of performing all 7 tasks on the GRIT benchmark and
produces strong results across 16 diverse benchmarks like NYUv2-Depth,
ImageNet, VQA2.0, OK-VQA, Swig, VizWizGround, BoolQ, and SciTail, with no
task-specific fine-tuning. Code and demos for Unified-IO are available at:
this https URL

Unified-IO 是一种模型，它可以执行各种人工智能任务，包括传统的计算机视觉任务、视觉与语言任务和自然语言处理任务，通过将每个任务的异构输入和输出转换为离散词汇令其有共同的表示方式，从而实现使用一个单一的基于 transformer 的架构，针对视觉和语言领域的 90 多个不同数据集联合训练，其能够在诸如 NYUv2-Depth、ImageNet、VQA2.0 等 16 个不同基准测试中产生强大的结果，这是第一个不需要针对特定任务进行微调即可执行所有 7 个 GRIT 基准任务的模型。