Defining Success Metrics across Generative and Agentic Experiences

About this session

When an AI agent completes a multi-step task, how do you actually know it succeeded? As generative and agentic experiences move from demos into production, most teams are still applying metrics built for deterministic software, and those metrics quietly miss what these systems get wrong. I’ll draw on my experience shipping AI products at Amazon to show where those metrics break down. I’ll share a practical framework for defining success across generative outputs and multi-step agent workflows, balancing accuracy, user trust, and business impact. You will leave with a concrete set of questions to run against their own product before it ships.