Beyond Benchmarks: The Lifecycle of Measuring Agentic Quality in AI Content Management

By Sujay Kumar Jauhar, Spriha Chandrayan, Natasha Gaitonde, Zhen Lei, Reed Pankhurst, Anush Sankaran, Amrit Shandilya, Ryen W. White Organizations put their most important content in OneDrive and SharePoint, trusting us to deliver AI experiences they can depend on. That trust run…

By Sujay Kumar Jauhar, Spriha Chandrayan, Natasha Gaitonde, Zhen Lei, Reed Pankhurst, Anush Sankaran, Amrit Shandilya, Ryen W. White Organizations put their most important content in OneDrive and SharePoint, trusting us to deliver AI experiences they can depend on. That trust runs across everything Copilot does with their content — surfacing the right information through retrieval, and, increasingly, driving agentic workflows that reason over documents and take action on their behalf. Living up to that trust is what we care about most, and it's why we put the quality of these experiences — and consequently the ways we measure that quality — at the center of how we build. It turns out that this is a non-trivial problem, because quality in AI is rarely binary. An agent can call exactly the r