Complex SOPs, Brittle Agents: Evaluating AI Agents in Real-World Automations

About this session

Real world standard operating procedures are complex, involving branching logic, long sequential steps, ordering constraints, rollback paths and ambiguous tool definitions. Current AI agents are surprisingly brittle when executing them and measuring them with task success rate is insufficient. As agents move from copilots to autonomous operators, this is a blind spot with real production consequences. This talk introduces an evaluation framework with comprehensive error taxonomy followed by empirical findings from evaluating frontier models across multiple domains. Attendees will leave with understanding of what makes real world AI agents challenging and practical approaches for building and evaluating these AI agents in real world.