Who Watches the Agents? Building Gen AI Evaluation Suites for Complex Architectures

About this session

While the industry spent the last year racing to build autonomous multi-agent systems, a critical question remains: how do we actually trust them in production? As enterprise adoption scales, the bottleneck has definitively shifted from agent creation to rigorous evaluation. In this talk, I will explore the practical realities of testing complex architectures where traditional LLM benchmarks fall short. I will demonstrate how to design robust evaluation suites for dynamic reasoning and tool use, specifically focusing on safeguarding sensitive workflows like strict OAuth integrations and data retrievals. You will also learn practical techniques for utilizing tools like the Vertex AI Prompt Optimizer to continuously refine agent behavior. Join me to discover how to move your multi-agent workflows out of the prototype phase and into secure, measurable enterprise production.