The dynamic nature of esports makes the situation relatively complicated for
average viewers. Esports broadcasting involves game expert casters, but the
caster-dependent game commentary is not enough to fully understand the game
situation. It will be richer by including diverse multimodal esports
information, including audiences' talks/emotions, game audio, and game match
event information. This paper introduces GAME-MUG, a new multimodal game
situation understanding and audience-engaged commentary generation dataset and
its strong baseline. Our dataset is collected from 2020-2022 LOL game live
streams from YouTube and Twitch, and includes multimodal esports game
information, including text, audio, and time-series event logs, for detecting
the game situation. In addition, we also propose a new audience conversation
augmented commentary dataset by covering the game situation and audience
conversation understanding, and introducing a robust joint multimodal dual
learning model as a baseline. We examine the model's game situation/event
understanding ability and commentary generation capability to show the
effectiveness of the multimodal aspects coverage and the joint integration
learning approach.

介绍 GAME-MUG，一个包含多模态游戏情境理解和观众参与评论生成数据集的新数据集，以及使用鲁棒性联合多模态双学习模型作为基线的新观众对话增强评论数据集。通过覆盖游戏情境和观众对话的学习，引入时间序列事件日志，检查模型对游戏情境 / 事件的理解能力和评论生成能力，展示多模态方面覆盖和联合集成学习方法的有效性。