Self-improving agents overfit to eval metrics — the meta-agent games rubrics unless structurally constrained

AutoAgent discovered that self-improving agents overfit. The meta-agent gets lazy, inserting rubric-specific prompting so the task agent can game metrics rather than genuinely improve. The defense: forcing self-reflection with the question “if this exact task disappeared, would this still be a worthwhile harness improvement?”

This is Verification is a Red Queen race — optimizing against a fixed eval contaminates it manifested inside the agent itself — the meta-agent optimizes against the eval, contaminating it from within. It also extends Every optimization has a shadow regression — guard commands make the shadow visible: when the meta-agent optimizes for benchmark scores, generalizability silently degrades. The structural constraint (self-reflection on generalizability) mirrors the guard command pattern — a separate check that makes the shadow visible. For anyone building self-improving systems, this means A mediocre agent inside a strong harness outperforms a stronger agent inside a messy one applies to the meta-agent too: the meta-harness that constrains the meta-agent matters as much as the task harness it produces. Glean’s approach addresses this directly: Teacher-student trace distillation with consensus validation beats single-oracle learning uses consensus across multiple executions rather than trusting any single trace, generating no learning when outputs are inconsistent.