As language agents progressively automate critical tasks across domains, their ability to operate within operational constraints and safety protocols becomes essential. While extensive research has demonstrated these agents' effectiveness in downstream task completion, their reliability in following operational procedures and constraints remains largely unexplored. To this end, we present AgentOrca, a dual-system framework for evaluating language agents' compliance with operational constraints and routines. Our framework encodes action constraints and routines through both natural language prompts for agents and corresponding executable code serving as ground truth for automated verification. Through an automated pipeline of test case generation and evaluation across five real-world domains, we quantitatively assess current language agents' adherence to operational constraints. Our findings reveal notable performance gaps among state-of-the-art models, with large reasoning models like o1 demonstrating superior compliance while others show significantly lower performance, particularly when encountering complex constraints or user persuasion attempts.

本研究针对语言代理在遵循操作程序和约束方面的可靠性不足问题，提出了AgentOrca框架。该框架通过自然语言提示和可执行代码编码操作约束及程序，采用自动化的测试用例生成与评估方法，量化分析当前语言代理的遵循情况。研究显示，先进模型在遵循操作约束方面存在显著的性能差距，尤其是在复杂约束或用户劝说尝试时表现不佳。

AgentOrca：评估语言代理在操作规范和约束遵循方面的双系统框架