The Natural Language Processing(NLP) community has been using crowd sourcing
techniques to create benchmark datasets such as General Language Understanding
and Evaluation(GLUE) for training modern Language Models such as BERT. GLUE
tasks measure the reliability scores using inter annotator metrics i.e. Cohens
Kappa. However, the reliability aspect of LMs has often been overlooked. To
counter this problem, we explore a knowledge-guided LM ensembling approach that
leverages reinforcement learning to integrate knowledge from ConceptNet and
Wikipedia as knowledge graph embeddings. This approach mimics human annotators
resorting to external knowledge to compensate for information deficits in the
datasets. Across nine GLUE datasets, our research shows that ensembling
strengthens reliability and accuracy scores, outperforming state of the art.

利用强化学习整合来自 ConceptNet 和维基百科的知识图嵌入，以提高语言模型的可靠性和准确性，在九个 GLUE 数据集上表现优于现有方法。

借助知识和增强学习提升语言模型的可靠性

Leveraging Knowledge and Reinforcement Learning for Enhanced Reliability  of Language Models

This paper deals with Audio-Visual Speech Recognition (AVSR) under multimodal
input corruption situations where audio inputs and visual inputs are both
corrupted, which is not well addressed in previous research directions.
Previous studies have focused on how to complement the corrupted audio inputs
with the clean visual inputs with the assumption of the availability of clean
visual inputs. However, in real life, clean visual inputs are not always
accessible and can even be corrupted by occluded lip regions or noises. Thus,
we firstly analyze that the previous AVSR models are not indeed robust to the
corruption of multimodal input streams, the audio and the visual inputs,
compared to uni-modal models. Then, we design multimodal input corruption
modeling to develop robust AVSR models. Lastly, we propose a novel AVSR
framework, namely Audio-Visual Reliability Scoring module (AV-RelScore), that
is robust to the corrupted multimodal inputs. The AV-RelScore can determine
which input modal stream is reliable or not for the prediction and also can
exploit the more reliable streams in prediction. The effectiveness of the
proposed method is evaluated with comprehensive experiments on popular
benchmark databases, LRS2 and LRS3. We also show that the reliability scores
obtained by AV-RelScore well reflect the degree of corruption and make the
proposed model focus on the reliable multimodal representations.

本文针对音频和视频同时受损的多模态输入情况下的视音频说话人识别问题展开研究，通过分析现有模型的不足并引入多模态输入损坏模型来设计一个稳健的 AVSR 模型框架，即 AV-RelScore，通过可靠性得分确定可靠输入流并提高识别准确度。