Multimodal data analysis and validation based on streams from
state-of-the-art sensor technology such as eye-tracking or emotion recognition
using the Facial Action Coding System (FACTs) with deep learning allows
educational researchers to study multifaceted learning and problem-solving
processes and to improve educational experiences. This study aims to
investigate the correlation between two continuous sensor streams, pupil
diameter as an indicator of cognitive workload and FACTs with deep learning as
an indicator of emotional arousal (RQ 1a), specifically for epochs of high,
medium, and low arousal (RQ 1b). Furthermore, the time lag between emotional
arousal and pupil diameter data will be analyzed (RQ 2). 28 participants worked
on three cognitively demanding and emotionally engaging everyday moral dilemmas
while eye-tracking and emotion recognition data were collected. The data were
pre-processed in Phyton (synchronization, blink control, downsampling) and
analyzed using correlation analysis and Granger causality tests. The results
show negative and statistically significant correlations between the data
streams for emotional arousal and pupil diameter. However, the correlation is
negative and significant only for epochs of high arousal, while positive but
non-significant relationships were found for epochs of medium or low arousal.
The average time lag for the relationship between arousal and pupil diameter
was 2.8 ms. In contrast to previous findings without a multimodal approach
suggesting a positive correlation between the constructs, the results
contribute to the state of research by highlighting the importance of
multimodal data validation and research on convergent vagility. Future research
should consider emotional regulation strategies and emotional valence.

通过多模态数据分析和基于最先进的传感器技术（如眼动跟踪和面部行为编码系统）的数据验证，教育研究人员可以研究多方面的学习和问题解决过程，以及改善教育经验。本研究旨在调查两种连续传感器流（瞳孔直径作为认知负荷指标和基于深度学习的面部行为编码系统作为情绪激活指标）之间的相关性，并分析情绪激活和瞳孔直径数据之间的时间延迟。结果显示情绪激活和瞳孔直径数据之间存在负相关，相关性仅在高情绪激活阶段为负相关，而在中等或低情绪激活阶段存在正相关但不显著。未来研究应考虑情绪调节策略和情感价值。

通过多模态测量探索认知负荷和情绪激发的动态相互作用：瞳孔直径与情绪激发在情感参与任务中的相关性

Exploring the dynamic interplay of cognitive load and emotional arousal  by using multimodal measurements: Correlation of pupil diameter and emotional  arousal in emotionally engaging tasks

This study aims to explore the complex relationship between perceptual and
cognitive interactions in multimodal data analysis,with a specific emphasis on
spatial experience design in overseas Chinese gardens. It is found that
evaluation content and images on social media can reflect individuals' concerns
and sentiment responses, providing a rich data base for cognitive research that
contains both sentimental and image-based cognitive information. Leveraging
deep learning techniques, we analyze textual and visual data from social media,
thereby unveiling the relationship between people's perceptions and sentiment
cognition within the context of overseas Chinese gardens. In addition, our
study introduces a multi-agent system (MAS)alongside AI agents. Each agent
explores the laws of aesthetic cognition through chat scene simulation combined
with web search. This study goes beyond the traditional approach of translating
perceptions into sentiment scores, allowing for an extension of the research
methodology in terms of directly analyzing texts and digging deeper into
opinion data. This study provides new perspectives for understanding aesthetic
experience and its impact on architecture and landscape design across diverse
cultural contexts, which is an essential contribution to the field of cultural
communication and aesthetic understanding.

本研究探索了多模态数据分析中感知和认知相互作用之间的复杂关系，特别强调了海外华人园林的空间体验设计。通过分析社交媒体中的评价内容和图像，发现其可以反映个体的关注和情感反应，为认知研究提供了富含情感和图像认知信息的数据基础。利用深度学习技术，我们分析社交媒体中的文本和视觉数据，揭示了海外华人园林背景下人们感知和情感认知之间的关系。此外，本研究引入多智能体系统 (MAS) 和 AI 智能体，通过对话场景模拟和网络搜索，探索审美认知规律。本研究通过直接分析文本和深入挖掘意见数据，超越了将感知转化为情感评分的传统方法，为研究方法论的扩展提供了一种可能。该研究为了解跨不同文化背景下建筑与景观设计的审美体验及其影响提供了新的视角，对文化传播和审美理解领域具有重要贡献。