Live commenting on video, a popular feature of live streaming platforms,
enables viewers to engage with the content and share their comments, reactions,
opinions, or questions with the streamer or other viewers while watching the
video or live stream. It presents a challenging testbed for AI agents, which
involves the simultaneous understanding of audio-visual multimodal contexts
from live streams and the ability to interact with human viewers through
dialogue. As existing live streaming-based comments datasets contain limited
categories and lack a diversity, we create a large-scale audio-visual
multimodal dialogue dataset to facilitate the development of live commenting
technologies. The data is collected from Twitch, with 11 different categories
and 575 streamers for a total of 438 hours of video and 3.2 million comments.
Moreover, we propose a novel multimodal generation model capable of generating
live comments that align with the temporal and spatial events within the video,
as well as with the ongoing multimodal dialogue context. Our initial results
have demonstrated the effectiveness of the proposed model, providing a robust
foundation for further research and practical applications in the field of live
video interaction.

通过创建大规模的音视频多模式对话数据集，以促进直播评论技术的发展，我们还提出了一种能够生成与视频中的时空事件以及正在进行的多模式对话上下文相吻合的实时评论的新颖多模式生成模型。

LiveChat: 从视听多模态环境生成视频评论

LiveChat: Video Comment Generation from Audio-Visual Multimodal Contexts

In recent years, live streaming platforms have gained immense popularity as
they allow users to broadcast their videos and interact in real-time with hosts
and peers. Due to the dynamic changes of live content, accurate recommendation
models are crucial for enhancing user experience. However, most previous works
treat the live as a whole item and explore the Click-through-Rate (CTR)
prediction framework on item-level, neglecting that the dynamic changes that
occur even within the same live room. In this paper, we proposed a ContentCTR
model that leverages multimodal transformer for frame-level CTR prediction.
First, we present an end-to-end framework that can make full use of multimodal
information, including visual frames, audio, and comments, to identify the most
attractive live frames. Second, to prevent the model from collapsing into a
mediocre solution, a novel pairwise loss function with first-order difference
constraints is proposed to utilize the contrastive information existing in the
highlight and non-highlight frames. Additionally, we design a temporal
text-video alignment module based on Dynamic Time Warping to eliminate noise
caused by the ambiguity and non-sequential alignment of visual and textual
information. We conduct extensive experiments on both real-world scenarios and
public datasets, and our ContentCTR model outperforms traditional
recommendation models in capturing real-time content changes. Moreover, we
deploy the proposed method on our company platform, and the results of online
A/B testing further validate its practical significance.

本文提出了一种基于多模态 Transformer 的 ContentCTR 模型，采用帧级别的 CTR 预测来充分利用视觉帧、音频和评论等多模态信息，借助具有一阶差分约束的新型成对损失函数来利用亮点和非亮点间的对比信息，以及基于动态时间规整的时间文本 - 视频对齐模块以捕捉实时内容变化，实验表明该方法在真实场景和公共数据集上进行推荐时均优于传统方法，应用于公司平台后的在线 A/B 测试也进一步验证了其实际意义。