Multimodal corpora have become an essential language resource for language
science and grounded natural language processing (NLP) systems due to the
growing need to understand and interpret human communication across various
channels. In this paper, we first present our efforts in building the first
Multimodal Corpus for Languages in Taiwan (MultiMoco). Based on the corpus, we
conduct a case study investigating the Lexical Retrieval Hypothesis (LRH),
specifically examining whether the hand gestures co-occurring with speech
constants facilitate lexical retrieval or serve other discourse functions. With
detailed annotations on eight parliamentary interpellations in Taiwan Mandarin,
we explore the co-occurrence between speech constants and non-verbal features
(i.e., head movement, face movement, hand gesture, and function of hand
gesture). Our findings suggest that while hand gestures do serve as
facilitators for lexical retrieval in some cases, they also serve the purpose
of information emphasis. This study highlights the potential of the MultiMoco
Corpus to provide an important resource for in-depth analysis and further
research in multimodal communication studies.

该论文描述了建立台湾语言的第一个多模态语料库 (MultiMoco) 的努力，并通过研究手势和语音恒速器的共现，探讨了词汇检索假说的有效性，结果表明手势在某些情况下确实有助于词汇检索，同时也用于信息强调。

多模态语境下的词汇检索假设

Lexical Retrieval Hypothesis in Multimodal Context

Human language is firstly spoken and only secondarily written.
Text, however, is a very convenient and efficient
representation of language, and modern civilization has made it
ubiquitous. Thus the field of NLP has
overwhelmingly focused on processing written rather than spoken
language. Work on spoken language, on the other hand, has been
siloed off within the largely separate speech processing community
which has been inordinately preoccupied with transcribing speech into
text.
Recent advances in deep learning have led to a fortuitous
convergence in methods between speech processing and mainstream NLP.
Arguably, the time is ripe for a unification of these two fields,
and for starting to take spoken language seriously as the primary
mode of human communication. Truly natural language processing
could lead to better integration with the rest of language science
and could lead to systems which are more data-efficient and more
human-like, and which can communicate beyond the textual
modality.

利用深度学习，将口语处理和主流自然语言处理结合起来，以实现更符合人类交流方式、更具数据效率和更具人类化的系统，并更好地与其他语言科学进行整合。