Viewport prediction is a crucial aspect of tile-based 360 video streaming system. However, existing trajectory based methods lack of robustness, also oversimplify the process of information construction and fusion between different modality inputs, leading to the error accumulation problem. In this paper, we propose a tile classification based viewport prediction method with Multi-modal Fusion Transformer, namely MFTR. Specifically, MFTR utilizes transformer-based networks to extract the long-range dependencies within each modality, then mine intra- and inter-modality relations to capture the combined impact of user historical inputs and video contents on future viewport selection. In addition, MFTR categorizes future tiles into two categories: user interested or not, and selects future viewport as the region that contains most user interested tiles. Comparing with predicting head trajectories, choosing future viewport based on tile's binary classification results exhibits better robustness and interpretability. To evaluate our proposed MFTR, we conduct extensive experiments on two widely used PVS-HM and Xu-Gaze dataset. MFTR shows superior performance over state-of-the-art methods in terms of average prediction accuracy and overlap ratio, also presents competitive computation efficiency.

提出了一种基于瓷砖分类的视口预测方法，使用多模态融合变压器提取每种模态的长距离依赖关系，从而捕捉用户历史输入和视频内容对未来视口选择的综合影响，并根据瓷砖的二分类结果选择未来视口，具有更好的鲁棒性和解释性。在两个常用的PVS-HM和Xu-Gaze数据集上进行了广泛实验，MFTR表现出卓越的性能，包括平均预测准确率和重叠率，同时展现出有竞争力的计算效率。

基于多模态融合变换器的瓷砖分类视口预测