Aligning machine learning systems with human expectations is mostly attempted
by training with manually vetted human behavioral samples, typically explicit
feedback. This is done on a population level since the context that is
capturing the subjective Point-Of-View (POV) of a concrete person in a specific
situational context is not retained in the data. However, we argue that
alignment on an individual level can boost the subjective predictive
performance for the individual user interacting with the system considerably.
Since perception differs for each person, the same situation is observed
differently. Consequently, the basis for decision making and the subsequent
reasoning processes and observable reactions differ. We hypothesize that
individual perception patterns can be used for improving the alignment on an
individual level. We test this, by integrating perception information into
machine learning systems and measuring their predictive performance
wrt.~individual subjective assessments. For our empirical study, we collect a
novel data set of multimodal stimuli and corresponding eye tracking sequences
for the novel task of Perception-Guided Crossmodal Entailment and tackle it
with our Perception-Guided Multimodal Transformer. Our findings suggest that
exploiting individual perception signals for the machine learning of subjective
human assessments provides a valuable cue for individual alignment. It does not
only improve the overall predictive performance from the point-of-view of the
individual user but might also contribute to steering AI systems towards every
person's individual expectations and values.

通过整合个体感知信号来提高个体用户与机器学习系统之间的相互适应对个体预测性能具有宝贵的启示，不仅可以提高个体用户的整体预测性能，还可以推动人工智能系统朝着每个人的个性化期望和价值导向发展。