Action quality assessment (AQA) is to assess how well an action is performed.
Previous works perform modelling by only the use of visual information,
ignoring audio information. We argue that although AQA is highly dependent on
visual information, the audio is useful complementary information for improving
the score regression accuracy, especially for sports with background music,
such as figure skating and rhythmic gymnastics. To leverage multimodal
information for AQA, i.e., RGB, optical flow and audio information, we propose
a Progressive Adaptive Multimodal Fusion Network (PAMFN) that separately models
modality-specific information and mixed-modality information. Our model
consists of with three modality-specific branches that independently explore
modality-specific information and a mixed-modality branch that progressively
aggregates the modality-specific information from the modality-specific
branches. To build the bridge between modality-specific branches and the
mixed-modality branch, three novel modules are proposed. First, a
Modality-specific Feature Decoder module is designed to selectively transfer
modality-specific information to the mixed-modality branch. Second, when
exploring the interaction between modality-specific information, we argue that
using an invariant multimodal fusion policy may lead to suboptimal results, so
as to take the potential diversity in different parts of an action into
consideration. Therefore, an Adaptive Fusion Module is proposed to learn
adaptive multimodal fusion policies in different parts of an action. This
module consists of several FusionNets for exploring different multimodal fusion
strategies and a PolicyNet for deciding which FusionNets are enabled. Third, a
module called Cross-modal Feature Decoder is designed to transfer cross-modal
features generated by Adaptive Fusion Module to the mixed-modality branch.

通过提出一种渐进自适应多模态融合网络（PAMFN），该网络利用视觉信息和音频信息相结合以提高动作质量评估的准确性。研究表明，音频是一种有效的补充信息，尤其对于带有背景音乐的体育项目，如花样滑冰和韵律体操。PAMFN 由三个专注于各自信息的分支和一个逐渐聚合各种信息的混合分支构成，采用了一些创新模块来建立各分支之间的联系。这个研究还引入了自适应融合模块，旨在学习不同动作部分的适应性多模态融合策略。

多模态行动质量评估

Multimodal Action Quality Assessment

Aerial scene recognition is a fundamental task in remote sensing and has
recently received increased interest. While the visual information from
overhead images with powerful models and efficient algorithms yields
considerable performance on scene recognition, it still suffers from the
variation of ground objects, lighting conditions etc. Inspired by the
multi-channel perception theory in cognition science, in this paper, for
improving the performance on the aerial scene recognition, we explore a novel
audiovisual aerial scene recognition task using both images and sounds as
input. Based on an observation that some specific sound events are more likely
to be heard at a given geographic location, we propose to exploit the knowledge
from the sound events to improve the performance on the aerial scene
recognition. For this purpose, we have constructed a new dataset named AuDio
Visual Aerial sceNe reCognition datasEt (ADVANCE). With the help of this
dataset, we evaluate three proposed approaches for transferring the sound event
knowledge to the aerial scene recognition task in a multimodal learning
framework, and show the benefit of exploiting the audio information for the
aerial scene recognition. The source code is publicly available for
reproducibility purposes.

利用声音信息结合图像信息进行空中场景识别，通过构建新的 AVANET 数据集，通过多模式的学习框架将声音事件的知识转化到空中场景识别的任务中，提高识别效果。