This technical report describes our QuAVF@NTU-NVIDIA submission to the Ego4D
Talking to Me (TTM) Challenge 2023. Based on the observation from the TTM task
and the provided dataset, we propose to use two separate models to process the
input videos and audio. By doing so, we can utilize all the labeled training
data, including those without bounding box labels. Furthermore, we leverage the
face quality score from a facial landmark prediction model for filtering noisy
face input data. The face quality score is also employed in our proposed
quality-aware fusion for integrating the results from two branches. With the
simple architecture design, our model achieves 67.4% mean average precision
(mAP) on the test set, which ranks first on the leaderboard and outperforms the
baseline method by a large margin. Code is available at:
this https URL

本文介绍了 QuAVF@NTU-NVIDIA 团队在 Ego4D Talking to Me (TTM) Challenge 2023 中的参赛作品，该作品利用两个模型处理输入视频和语音，通过面部质量分数过滤嘈杂的面部输入数据并实现质量感知融合，取得了 67.4％的均值精度 (mAP) 的最优结果。