Traditional human-in-the-loop-based annotation for time-series data like
inertial data often requires access to alternate modalities like video or audio
from the environment. These alternate sources provide the necessary information
to the human annotator, as the raw numeric data is often too obfuscated even
for an expert. However, this traditional approach has many concerns surrounding
overall cost, efficiency, storage of additional modalities, time, scalability,
and privacy. Interestingly, recent large language models (LLMs) are also
trained with vast amounts of publicly available alphanumeric data, which allows
them to comprehend and perform well on tasks beyond natural language
processing. Naturally, this opens up a potential avenue to explore LLMs as
virtual annotators where the LLMs will be directly provided the raw sensor data
for annotation instead of relying on any alternate modality. Naturally, this
could mitigate the problems of the traditional human-in-the-loop approach.
Motivated by this observation, we perform a detailed study in this paper to
assess whether the state-of-the-art (SOTA) LLMs can be used as virtual
annotators for labeling time-series physical sensing data. To perform this in a
principled manner, we segregate the study into two major phases. In the first
phase, we investigate the challenges an LLM like GPT-4 faces in comprehending
raw sensor data. Considering the observations from phase 1, in the next phase,
we investigate the possibility of encoding the raw sensor data using SOTA SSL
approaches and utilizing the projected time-series data to get annotations from
the LLM. Detailed evaluation with four benchmark HAR datasets shows that
SSL-based encoding and metric-based guidance allow the LLM to make more
reasonable decisions and provide accurate annotations without requiring
computationally expensive fine-tuning or sophisticated prompt engineering.

传统的基于人机协同的时间序列数据标注方法常常需要访问环境中的其他模态，例如视频或音频，以便为人类标注员提供必要的信息，因为原始数字数据常常对专家来说过于模糊。然而，这种传统方法在成本、效率、存储额外模态、时间、可扩展性和隐私等方面存在诸多问题。通过近期的大型语言模型（LLMs）的训练，这些模型还可以理解并在自然语言处理之外的任务上表现出色，这为将 LLMs 作为虚拟标注员进行探索开辟了潜在途径，其中 LLMs 将直接为标注提供原始传感器数据，而无需依赖任何其他模态。这自然地可以缓解传统的人机协同方法所面临的问题。受这一观察的启发，我们在本文中进行了一项详细研究，以评估最先进的 LLMs 是否可用作对时间序列物理感知数据进行标注的虚拟标注员。为了以系统的方式进行，我们将研究分为两个主要阶段。在第一阶段中，我们研究了 LLM（如 GPT-4）在理解原始传感器数据方面所面临的挑战。考虑到第一阶段的观察结果，在接下来的阶段中，我们研究了使用最先进的 SSL 方法对原始传感器数据进行编码的可能性，并利用映射后的时间序列数据从 LLM 获取标注。使用四个基准 HAR 数据集进行详细评估表明，基于 SSL 的编码和度量指导能够使 LLM 做出更合理的决策，并提供准确的标注，而无需进行计算昂贵的微调或复杂的提示工程。