'Actions' play a vital role in how humans interact with the world. Thus,
autonomous agents that would assist us in everyday tasks also require the
capability to perform 'Reasoning about Actions & Change' (RAC). This has been
an important research direction in Artificial Intelligence (AI) in general, but
the study of RAC with visual and linguistic inputs is relatively recent. The
CLEVR_HYP (Sampat et. al., 2021) is one such testbed for hypothetical
vision-language reasoning with actions as the key focus. In this work, we
propose a novel learning strategy that can improve reasoning about the effects
of actions. We implement an encoder-decoder architecture to learn the
representation of actions as vectors. We combine the aforementioned
encoder-decoder architecture with existing modality parsers and a scene graph
question answering model to evaluate our proposed system on the CLEVR_HYP
dataset. We conduct thorough experiments to demonstrate the effectiveness of
our proposed approach and discuss its advantages over previous baselines in
terms of performance, data efficiency, and generalization capability.

通过使用编码器 - 解码器架构、模态分析器和场景图问题解答模型，我们提出了一种新的学习策略，可以提高关于行动效果的推理能力，并在 CLEVR_HYP 数据集上对我们的系统进行了评估。