Lexical semantics and cognitive science point to affordances (i.e. the
actions that objects support) as critical for understanding and representing
nouns and verbs. However, study of these semantic features has not yet been
integrated with the "foundation" models that currently dominate language
representation research. We hypothesize that predictive modeling of object
state over time will result in representations that encode object affordance
information "for free". We train a neural network to predict objects'
trajectories in a simulated interaction and show that our network's latent
representations differentiate between both observed and unobserved affordances.
We find that models trained using 3D simulations from our SPATIAL dataset
outperform conventional 2D computer vision models trained on a similar task,
and, on initial inspection, that differences between concepts correspond to
expected features (e.g., roll entails rotation). Our results suggest a way in
which modern deep learning approaches to grounded language learning can be
integrated with traditional formal semantic notions of lexical representations.

借助神经网络建模物体状态，可以实现有效的物体功能信息提取并与传统语义学模型相结合，能够在基于图像的语言学习上取得更好的表现。

以交互为预训练的地面功能表示学习

Pretraining on Interactions for Learning Grounded Affordance Representations

Modeling a structured, dynamic environment like a video game requires keeping
track of the objects and their states declarative knowledge) as well as
predicting how objects behave (procedural knowledge). Black-box models with a
monolithic hidden state often fail to apply procedural knowledge consistently
and uniformly, i.e., they lack systematicity. For example, in a video game,
correct prediction of one enemy's trajectory does not ensure correct prediction
of another's. We address this issue via an architecture that factorizes
declarative and procedural knowledge and that imposes modularity within each
form of knowledge. The architecture consists of active modules called object
files that maintain the state of a single object and invoke passive external
knowledge sources called schemata that prescribe state updates. To use a video
game as an illustration, two enemies of the same type will share schemata but
will have separate object files to encode their distinct state (e.g., health,
position). We propose to use attention to determine which object files to
update, the selection of schemata, and the propagation of information between
object files. The resulting architecture is a drop-in replacement conforming to
the same input-output interface as normal recurrent networks (e.g., LSTM, GRU)
yet achieves substantially better generalization on environments that have
multiple object tokens of the same type, including a challenging intuitive
physics benchmark.

该研究提出了一种基于对象状态和过程知识的模块化体系结构，使用注意力机制来管理不同对象的状态和更新规则，能够在视频游戏建模等领域具有更好的泛化性能。