Principle AI Product Building Architecture AI Agents

Private evals should measure business outcomes that matter — not external benchmarks

A firm's learning loop runs on private evals tied to real business outcomes and private RL environments trained on internal traces, so the model improves against what the company cares about rather than public leaderboards

@satyanadella (Satya Nadella) — A frontier without an ecosystem is not stable · Jun 15, 2026 · 6 connections

Nadella specifies the machinery a firm needs to turn workflows and judgment into improving AI: “Private evals should capture whether a model is actually improving against outcomes that matter to the business (not just external benchmarks!). Private reinforcement learning environments should let models grow stronger on real traces from inside the organization.” The knowledge base then “makes institutional memory queryable and use of tokens more efficient.” Public benchmarks measure generic capability; private evals measure whether the system is getting better at your outcomes.

This is the firm-level instantiation of treating Evals are behavioral pressure vectors, not neutral measurements — poorly chosen evals distort agent development — what you measure privately is what your system optimizes toward. It runs on the same substrate as Systems of record keep what was decided, not why — capture the reasoning at the moment a decision becomes binding and is why Give an optimizing agent full traces, not compressed summaries — it has to see why past attempts failed: the internal traces are both the RL signal and the institutional memory. Wired together, private evals plus internal-trace RL are the engine that powers the The trace→eval→harness flywheel compounds agent quality — every production interaction generates its own training data.

Connected Insights

References (4)

→ Systems of record keep what was decided, not why — capture the reasoning at the moment a decision becomes binding → Evals are behavioral pressure vectors, not neutral measurements — poorly chosen evals distort agent development → The trace→eval→harness flywheel compounds agent quality — every production interaction generates its own training data → Give an optimizing agent full traces, not compressed summaries — it has to see why past attempts failed

Referenced by (2)

← The trace→eval→harness flywheel compounds agent quality — every production interaction generates its own training data ← The quality gate should be a second adversarial model, not a human