Although domestic service robots are expected to assist individuals who
require support, they cannot currently interact smoothly with people through
natural language. For example, given the instruction "Bring me a bottle from
the kitchen," it is difficult for such robots to specify the bottle in an
indoor environment. Most conventional models have been trained on real-world
datasets that are labor-intensive to collect, and they have not fully leveraged
simulation data through a transfer learning framework. In this study, we
propose a novel transfer learning approach for multimodal language
understanding called Prototypical Contrastive Transfer Learning (PCTL), which
uses a new contrastive loss called Dual ProtoNCE. We introduce PCTL to the task
of identifying target objects in domestic environments according to free-form
natural language instructions. To validate PCTL, we built new real-world and
simulation datasets. Our experiment demonstrated that PCTL outperformed
existing methods. Specifically, PCTL achieved an accuracy of 78.1%, whereas
simple fine-tuning achieved an accuracy of 73.4%.

本研究提出一种新的转移学习方法，名为 PCTL，用于家庭环境中的语言理解任务，通过对比学习，使用对比损失 Dual ProtoNCE，利用真实世界和模拟数据集来验证 PCTL 的有效性，并实现了 78.1% 的精度。

多模态语言理解的原型对比迁移学习

Prototypical Contrastive Transfer Learning for Multimodal Language  Understanding

One common trend in recent studies of language models (LMs) is the use of
standardized tests for evaluation. However, despite being the fifth most spoken
language worldwide, few such evaluations have been conducted in Portuguese.
This is mainly due to the lack of high-quality datasets available to the
community for carrying out evaluations in Portuguese. To address this gap, we
introduce the Brazilian Leading Universities Entrance eXams (BLUEX), a dataset
of entrance exams from the two leading universities in Brazil: UNICAMP and USP.
The dataset includes annotated metadata for evaluating the performance of NLP
models on a variety of subjects. Furthermore, BLUEX includes a collection of
recently administered exams that are unlikely to be included in the training
data of many popular LMs as of 2023. The dataset is also annotated to indicate
the position of images in each question, providing a valuable resource for
advancing the state-of-the-art in multimodal language understanding and
reasoning. We describe the creation and characteristics of BLUEX and establish
a benchmark through experiments with state-of-the-art LMs, demonstrating its
potential for advancing the state-of-the-art in natural language understanding
and reasoning in Portuguese. The data and relevant code can be found at
this https URL

BLUEX 数据集为评价自然语言理解和推理在葡萄牙语中的最先进技术提供了基准，并标注了图像在每个问题中的位置，是推进多模态语言理解和推理技术的宝贵资源。

BLUEX：基于巴西顶尖大学入学考试的基准测试

BLUEX: A benchmark based on Brazilian Leading Universities Entrance  eXams

Recent voice assistants are usually based on the cascade spoken language
understanding (SLU) solution, which consists of an automatic speech recognition
(ASR) engine and a natural language understanding (NLU) system. Because such
approach relies on the ASR output, it often suffers from the so-called ASR
error propagation. In this work, we investigate impacts of this ASR error
propagation on state-of-the-art NLU systems based on pre-trained language
models (PLM), such as BERT and RoBERTa. Moreover, a multimodal language
understanding (MLU) module is proposed to mitigate SLU performance degradation
caused by errors present in the ASR transcript. The MLU benefits from
self-supervised features learned from both audio and text modalities,
specifically Wav2Vec for speech and Bert/RoBERTa for language. Our MLU combines
an encoder network to embed the audio signal and a text encoder to process text
transcripts followed by a late fusion layer to fuse audio and text logits. We
found that the proposed MLU showed to be robust towards poor quality ASR
transcripts, while the performance of BERT and RoBERTa are severely
compromised. Our model is evaluated on five tasks from three SLU datasets and
robustness is tested using ASR transcripts from three ASR engines. Results show
that the proposed approach effectively mitigates the ASR error propagation
problem, surpassing the PLM models' performance across all datasets for the
academic ASR engine.

本文研究了基于现有自然语言理解模型的语音助手中由自动语音识别误差引起的性能损失，并提出一种多模态语言理解模块，利用自监督学习从语音和文本模态中获取特征，最终通过对音频信号和文本传输进行编码来获得更好的性能。