The ability to actively ground task instructions from an egocentric view is
crucial for AI agents to accomplish tasks or assist humans virtually. One
important step towards this goal is to localize and track key active objects
that undergo major state change as a consequence of human actions/interactions
to the environment without being told exactly what/where to ground (e.g.,
localizing and tracking the `sponge` in video from the instruction "Dip the
`sponge` into the bucket."). While existing works approach this problem from a
pure vision perspective, we investigate to which extent the textual modality
(i.e., task instructions) and their interaction with visual modality can be
beneficial. Specifically, we propose to improve phrase grounding models'
ability on localizing the active objects by: (1) learning the role of `objects
undergoing change` and extracting them accurately from the instructions, (2)
leveraging pre- and post-conditions of the objects during actions, and (3)
recognizing the objects more robustly with descriptional knowledge. We leverage
large language models (LLMs) to extract the aforementioned action-object
knowledge, and design a per-object aggregation masking technique to effectively
perform joint inference on object phrases and symbolic knowledge. We evaluate
our framework on Ego4D and Epic-Kitchens datasets. Extensive experiments
demonstrate the effectiveness of our proposed framework, which leads to>54%
improvements in all standard metrics on the TREK-150-OPE-Det localization +
tracking task, >7% improvements in all standard metrics on the TREK-150-OPE
tracking task, and >3% improvements in average precision (AP) on the Ego4D SCOD
task.

通过学习任务指令的对象信息、利用对象的动作前后条件和描述性知识，我们提出了一种改进短语定位模型的方法，通过大型语言模型来提取动作 - 对象知识，并设计了一种对象短语和符号知识的联合推理技术，实验证明我们的方法在各项指标上都取得了显著的改善。