Large language models (LLMs) are typically prompted to follow a single
instruction per inference call. In this work, we analyze whether LLMs also hold
the capability to handle multiple instructions simultaneously, denoted as
Multi-Task Inference. For this purpose, we introduce the MTI Bench(Multi-Task
Inference Benchmark), a comprehensive evaluation benchmark encompassing 5,000
instances across 25 tasks. Each task in the MTI Bench involves 2 to 3
sub-tasks. As expected, we first demonstrate that Multi-Task Inference reduces
the total inference time by 1.46 times in average since it does not require
multiple inference calls. Interestingly, contrary to the expectation that LLMs
would perform better when tasks are divided, we find that state-of-the-art
LLMs, such as Llama-2-Chat-70B and GPT-4, show up to 7.3% and 12.4% improved
performance with Multi-Task Inference compared to Single-Task Inference on the
MTI Bench. We release the MTI Bench dataset and our code at this link
this https URL

该研究分析了大型语言模型的多任务推理能力，并介绍了 MTI Bench 作为一个综合评估基准，证明多任务推理相对于单任务推理可以显著减少推理时间并提高性能。