Natural language understanding is one of the most challenging topics in
artificial intelligence. Deep neural network methods, particularly large
language module (LLM) methods such as ChatGPT and GPT-3, have powerful
flexibility to adopt informal text but are weak on logical deduction and suffer
from the out-of-vocabulary (OOV) problem. On the other hand, rule-based methods
such as Mathematica, Semantic web, and Lean, are excellent in reasoning but
cannot handle the complex and changeable informal text. Inspired by pragmatics
and structuralism, we propose two strategies to solve the OOV problem and a
semantic model for better natural language understanding and reasoning.

本文提出了两种解决 out-of-vocabulary 问题的策略和一个语义模型，以解决传统深度神经网络和基于规则的方法在自然语言理解上所面临的挑战。

元语义学：迈向更好的自然语言理解和推理

Meta Semantics: Towards better natural language understanding and  reasoning

Recently, pre-trained language models have achieved remarkable success in a
broad range of natural language processing tasks. However, in multilingual
setting, it is extremely resource-consuming to pre-train a deep language model
over large-scale corpora for each language. Instead of exhaustively
pre-training monolingual language models independently, an alternative solution
is to pre-train a powerful multilingual deep language model over large-scale
corpora in hundreds of languages. However, the vocabulary size for each
language in such a model is relatively small, especially for low-resource
languages. This limitation inevitably hinders the performance of these
multilingual models on tasks such as sequence labeling, wherein in-depth
token-level or sentence-level understanding is essential.
In this paper, inspired by previous methods designed for monolingual
settings, we investigate two approaches (i.e., joint mapping and mixture
mapping) based on a pre-trained multilingual model BERT for addressing the
out-of-vocabulary (OOV) problem on a variety of tasks, including part-of-speech
tagging, named entity recognition, machine translation quality estimation, and
machine reading comprehension. Experimental results show that using mixture
mapping is more promising. To the best of our knowledge, this is the first work
that attempts to address and discuss the OOV issue in multilingual settings.

本文研究了两种方法（联合映射和混合映射）来解决预先训练的 BERT 多语言模型中的词汇表大小和词汇缺失问题，并在多个任务中进行了实验。结果表明，使用混合映射更有前途。这是首次在多语言环境下试图解决词汇缺失问题。

使用词汇扩展提高预训练多语言模型的性能

Improving Pre-Trained Multilingual Models with Vocabulary Expansion

End-to-end task-oriented dialogue is challenging since knowledge bases are
usually large, dynamic and hard to incorporate into a learning framework. We
propose the global-to-local memory pointer (GLMP) networks to address this
issue. In our model, a global memory encoder and a local memory decoder are
proposed to share external knowledge. The encoder encodes dialogue history,
modifies global contextual representation, and generates a global memory
pointer. The decoder first generates a sketch response with unfilled slots.
Next, it passes the global memory pointer to filter the external knowledge for
relevant information, then instantiates the slots via the local memory
pointers. We empirically show that our model can improve copy accuracy and
mitigate the common out-of-vocabulary problem. As a result, GLMP is able to
improve over the previous state-of-the-art models in both simulated bAbI
Dialogue dataset and human-human Stanford Multi-domain Dialogue dataset on
automatic and human evaluation.

该论文提出了一种名为 GLMP 网络的机制，旨在帮助解决在端到端任务导向对话中，将大型动态的知识库融入到学习框架中存在的挑战，并且通过实验证明其在模拟 bAbI 对话数据集和 Stanford 多领域对话数据集上的样本和人工评估中，都表现得比先前最先进的模型更好。