Responsible AI after the demo: evaluating and observing agents in Microsoft Foundry

ESPC26
Conference Sessions

Copilot, Agents, and AI

Responsible AI after the demo: evaluating and observing agents in Microsoft Foundry

Developers

Level 200

SPEAKERS

Maarten Eekels

Microsoft Regional Director, MVP

ABOUT THE SESSION

Building an impressive AI agent is relatively easy. Proving that it remains useful, safe, reliable, and responsible once real users start interacting with it is much harder.

In this session, we explore how Microsoft Foundry helps teams turn Responsible AI principles into a continuous engineering discipline. We will look beyond traditional monitoring and show how tracing and observability expose what an agent actually did, how rubric-based and deterministic evaluations measure behavior against your own requirements, and why multi-turn evaluation is essential when agents use memory, tools, and conversational context.

But evaluating the agent is only part of the challenge. How do you know whether an AI-based evaluator can itself be trusted? How do you test agents against adversarial behavior? What happens when production traces contain sensitive information? And ultimately, how do you connect technical evaluation metrics such as groundedness and task adherence to actual business outcomes?

You will leave with a practical approach for moving from “the demo worked” to having measurable evidence that your agent remains trustworthy in production.

Assumed Knowledge:

You should have a basic understanding of generative AI, large language models, and how AI agents use prompts, context, and tools to complete tasks.

Practical Takeaways:

After this session, attendees will be able to:

1. Evaluate whether an AI agent behaves as intended using tracing, deterministic checks, Microsoft Foundry evaluators, custom rubrics, and multi-turn evaluation.

2. Turn Responsible AI principles into measurable engineering requirements including evaluation criteria, release thresholds, evaluator calibration, adversarial testing, and business outcome metrics.

Session Scope:

We will not build an AI agent from scratch and this session is not a theoretical introduction to AI ethics