Most studies on machine learning in sensing systems focus on low-level
perception tasks that process raw sensory data within a short time window.
However, many practical applications, such as human routine modeling and
occupancy tracking, require high-level reasoning abilities to comprehend
concepts and make inferences based on long-term sensor traces. Existing machine
learning-based approaches for handling such complex tasks struggle to
generalize due to the limited training samples and the high dimensionality of
sensor traces, necessitating the integration of human knowledge for designing
first-principle models or logic reasoning methods. We pose a fundamental
question: Can we harness the reasoning capabilities and world knowledge of
Large Language Models (LLMs) to recognize complex events from long-term
spatiotemporal sensor traces? To answer this question, we design an effective
prompting framework for LLMs on high-level reasoning tasks, which can handle
traces from the raw sensor data as well as the low-level perception results. We
also design two strategies to enhance performance with long sensor traces,
including summarization before reasoning and selective inclusion of historical
traces. Our framework can be implemented in an edge-cloud setup, running small
LLMs on the edge for data summarization and performing high-level reasoning on
the cloud for privacy preservation. The results show that LLMSense can achieve
over 80\% accuracy on two high-level reasoning tasks such as dementia diagnosis
with behavior traces and occupancy tracking with environmental sensor traces.
This paper provides a few insights and guidelines for leveraging LLM for
high-level reasoning on sensor traces and highlights several directions for
future work.

通过设计一个高效的提示框架，本研究利用大型语言模型（LLMs）的推理能力和世界知识，从长期的时空传感数据中识别复杂事件，并提供了一些关于在传感数据上利用 LLMs 进行高级推理的见解和指导，以及未来工作的几个方向。

LLMSense：利用 LLM 进行高级别时空传感器轨迹推理

LLMSense: Harnessing LLMs for High-level Reasoning Over Spatiotemporal  Sensor Traces

While human speakers use a variety of different expressions when describing
the same object in an image, giving rise to a distribution of plausible labels
driven by pragmatic constraints, the extent to which current Vision \& Language
Large Language Models (VLLMs) can mimic this crucial feature of language use is
an open question. This applies to common, everyday objects, but it is
particularly interesting for uncommon or novel objects for which a category
label may be lacking or fuzzy. Furthermore, humans show clear production
preferences for highly context-sensitive expressions, such as the quantifiers
`few' or `most'. In our work, we evaluate VLLMs (FROMAGe, BLIP-2, LLaVA) on
three categories (nouns, attributes, and quantifiers) where humans show great
subjective variability concerning the distribution over plausible labels, using
datasets and resources mostly under-explored in previous work. Our results
reveal mixed evidence on the ability of VLLMs to capture human naming
preferences, with all models failing in tasks that require high-level reasoning
such as assigning quantifiers.

当前的视觉与语言大型语言模型（VLLMs）在捕捉人类命名偏好方面的能力存在问 题，尤其在需要高级推理能力的任务中，例如指定量词。

人类和 LLMs 中的视觉对象命名、描述和量化

Naming, Describing, and Quantifying Visual Objects in Humans and LLMs

High-level reasoning can be defined as the capability to generalize over
knowledge acquired via experience, and to exhibit robust behavior in novel
situations. Such form of reasoning is a basic skill in humans, who seamlessly
use it in a broad spectrum of tasks, from language communication to decision
making in complex situations. When it manifests itself in understanding and
manipulating the everyday world of objects and their interactions, we talk
about common sense or commonsense reasoning. State-of-the-art AI systems don't
possess such capability: for instance, Large Language Models have recently
become popular by demonstrating remarkable fluency in conversing with humans,
but they still make trivial mistakes when probed for commonsense competence; on
a different level, performance degradation outside training data prevents
self-driving vehicles to safely adapt to unseen scenarios, a serious and
unsolved problem that limits the adoption of such technology. In this paper we
propose to enable high-level reasoning in AI systems by integrating cognitive
architectures with external neuro-symbolic components. We illustrate a hybrid
framework centered on ACT-R and we discuss the role of generative models in
recent and future applications.

我们提出通过将认知架构与外部神经符号组件相结合，使人工智能系统能够进行高层次推理，讨论生成模型在最近和未来应用中的作用。

使用认知神经符号系统实现高层机器推理

Enabling High-Level Machine Reasoning with Cognitive Neuro-Symbolic  Systems

The ability of generative language models (GLMs) to generate text has
improved considerably in the last few years, enabling their use for generative
data augmentation. In this work, we propose CONDA, an approach to further
improve GLMs' ability to generate synthetic data by reformulating data
generation as context generation for a given question-answer (QA) pair and
leveraging QA datasets for training context generators. Then, we cast
downstream tasks into the same question answering format and adapt the
fine-tuned context generators to the target task domain. Finally, we use the
fine-tuned GLM to generate relevant contexts, which are in turn used as
synthetic training data for their corresponding tasks. We perform extensive
experiments on multiple classification datasets and demonstrate substantial
improvements in performance for both few- and zero-shot settings. Our analysis
reveals that QA datasets that require high-level reasoning abilities (e.g.,
abstractive and common-sense QA datasets) tend to give the best boost in
performance in both few-shot and zero-shot settings.

本论文提出 CONDA 方法以生成虚拟数据，并通过将结构放在 QA 对上并使用 QA 数据集来训练上下文生成器来进一步提高生成语言模型的能力，通过将下游任务转化为相同的问答格式并调整上下文生成器适应目标任务域，使用精调的生成语言模型生成相关上下文，这些上下文作为相应任务的虚拟训练数据，在多个分类数据集上进行实验，显示出在几乎零样本和少样本设置中的性能显着改进，分析表明要求高级推理能力的 QA 数据集（例如，抽象和常识 QA 数据集）在几乎零样本和少样本设置中都能显着改进性能。