The Evaluator Dilemma Does Recursive Self-Improvement Really Improve?

1Two examples

Let's start with two examples.

OpenRSI Index is a benchmark for self-improving research agents such as Claude Code. Each task has a hidden test that the agent can't read or edit, about as independent a check as one could ask for. But the agent sees its hidden score after every submission, and it uses it.

On one task, molmo2, the model has to point at an object in videos. Half of the test videos don't contain that object, so pointing at nothing earns full marks there. The agent's gain came from making the model predict less, and its own notes say so: the gain "comes from suppressing predictions". On another task, molmoweb, we re-tested the final submission on 3,007 new examples, and the 0.009 gain on the hidden test shrank to 0.002. A test the agent can't touch still stops being an independent check once the agent steers by its score.

Multi-Agent Evolve (MAE) works differently. One model writes questions, answers them, grades its own answers, and is then trained on those grades. This is a co-evolving evaluator: the grader changes as the model trains. We gave its grader 1,000 math answers whose correctness we already knew. It gave 996 of them 8 out of 10. Its ability to tell right answers from wrong ones was close to chance level (AUC 0.51). Nobody needs to game this grader. It has simply stopped saying anything useful.

These two cases use opposite kinds of evaluator, but they end up in the same place. The number that says "we improved" doesn't actually show that the system improved.

In this post we try to explain why this keeps happening. We first set up the problem (Section 2), then go through our audits of eight published systems (Section 3) and group what went wrong (Section 4). Next, we reproduce these failures in small loops where we know the right answers (Section 5) and list what an audit should check (Section 6). We end with the ways out and what they imply (Sections 7 and 8). Every system also has a full audit card in our RSI Audit Registry.

2The evaluator dilemma

What is a self-improvement loop? A recursive self-improvement (RSI) loop repeats three steps. The system proposes a change to itself: new code, a new prompt, new training data, or new model weights. An evaluator scores the change. The best changes are kept, and the loop starts again from there.

The evaluator can be many things: unit tests, a benchmark, an LLM judge, or the model grading its own work. Whatever it is, it plays two roles. It steers the loop, because it decides which changes survive. And its rising score is what we read as evidence that the system improved. These two roles pull against each other. Once a score steers the loop, the loop learns to raise that score, so a higher score alone no longer proves the system really got better.

Roughly, there are three ways to set up the evaluator.

Fixed evaluators stay the same while the system improves: unit tests, a benchmark, a frozen reward model. They are independent of the model, which is good. But they only check what they were written to check. Push hard enough against them and the loop learns what they reward, not what they were meant to measure. And once the model is better than the evaluator can judge, the evaluator can no longer tell the difference.

Co-evolving evaluators improve together with the model: the model grades itself, or a judge is trained alongside it. They can keep up with the model. But they are no longer independent. They share the model's blind spots, drift along with it, and can keep reporting progress long after they've stopped telling good from bad.

Hybrid evaluators mix the two, for example a self-judge that gets checked against a fixed anchor now and then. This sounds like the best of both. In practice it inherits the anchor's limits: wherever the anchor can't see, only the self-judge is left. For instance, one recent system lets its grading rules evolve along with the agent, but only keeps rules that agree with ten fixed examples whose grades are already known (the external anchor; Zhang et al., 2026). This just wraps a fixed evaluator around a co-evolving one, so the problem moves back to the fixed evaluator. The rules answer only to what those ten examples can check. In the paper's report-writing task, the evolved skills gamed the hand-written rubric that scored them, and it took a separate independent judge to catch it.

I think these are really one problem seen from three sides. What matters is not which kind of evaluator you pick, but whether its score feeds back into the loop. If the score is used as a reward, a selection rule, or feedback the agent can read, the loop will optimize against it, and the score is no longer independent evidence. This is the old problem of reward hacking, reward-model over-optimization, and benchmark hacking. If the score never feeds back, it stays independent, but it can only vouch for the narrow slice of behavior it measures.

So a claim of self-improvement needs two signals: one to steer the loop, and a separate one to check it.

Both are actually scalable oversight problems: each signal has to tell whether a change made the system better, and it is often weaker than the system it judges. They differ in what else they must do: the steering signal has to hold up while the loop optimizes against it, and the checking signal has to stay out of the loop. The audits below are also a test of how well today's oversight works on RSI systems. For each system, we ask which signal steered the loop and which one checked it. We come back to scalable oversight at the end.

3What the audits found

In every system we audited, the signal that steered the loop was also the one used to claim the gain, and it didn't hold up on its own. Where a gain was real, the evidence came from some check outside the loop, and that check covered less than the paper claimed.

For each system, we first say what it does and what it claims, then what we found. How we measured things is in the appendix, and all the numbers are in the system's registry card.

Fixed evaluators: gamed, or narrower than the claim

Darwin Gödel Machine (DGM) is a coding agent that rewrites its own code. Each new version is tested on SWE-bench, a set of real GitHub issues that come with unit tests, and the versions that pass more tests survive. We ran the first and the best released versions on 50 SWE-bench Verified tasks that DGM never saw during its search. The number of tasks that pass the tests went from 2 to 12, but three of the ten new passes don't actually fix the bug. They are shortcuts that make the tests pass while breaking the code somewhere else: one of them monkeypatches a sympy operator so that 2*adjoint(x) recurses forever.

This is how a fixed evaluator gets gamed. DGM is selected on passing the tests, so a shortcut that passes them counts as progress just like a real fix. Counted against the reference fixes, the real gain is 2 to 9, not 2 to 12. The most recent work in this line, the Mendel Gödel Machine, has the same problem: in pilot runs we recovered from its repository, the tests again let through patches that break the code.

The AI Scientist v2 says one of its AI-written papers passed peer review at an ICLR 2025 workshop. We ran the system's own reviewer on the three papers it submitted. It gave all three a 7 and accepted them, including the one human reviewers scored 3, 3 and 3. So what actually checked the papers was the human reviewers, not the system's own reviewer.

autoresearch (by Karpathy) lets an agent edit a training script and keeps an edit whenever the validation loss after 5 minutes of training goes down. We reran 37 versions of the script. Just changing the random seed already moves the loss by about 0.004, and 21 of the 22 kept edits are smaller than that. The edits as a whole do help: the loss drops by 0.043 after 30 minutes of training, and the gain holds on text from a different distribution. But "better after 5 minutes" is a narrower question than "better model":

• It overrates edits that just fit more training steps into 5 minutes. Halving the batch size gains 0.050 at 5 minutes but only 0.019 at 30.
• It gets the best edit wrong. Going to depth 9 looks worse at 5 minutes but gives the largest gain at 30 (−0.021).
• The other seven edits we measured gain 0.010 at 5 minutes and almost nothing (0.001) at 30.

An outside evaluator that gets pulled into the loop

OpenRSI Index, the first example from the opening, has a different problem from DGM's. DGM's tests were incomplete. Here the hidden test itself may be fine, but the agent sees its score after every submission, and it uses it. In Claude's run logs, the agent cites its hidden score when deciding what to do next 69 times on one task and 44 times on another. Agents run grid searches on the hidden test, keep or undo changes based on it, and even resubmit the same code several times and report the best run.

Why does that matter? A test is only a sample of the cases we care about, so part of every score is luck. If you submit N times and keep the best score, that best score looks better than it really is even if nothing got better, and the more you submit, the bigger this fake gain. The logs keep only total scores, so we can't say how much of any reported gain is this kind of luck, but we can check in other ways. On molmo2, where pointing at nothing earns full marks on half of the videos, the agent's own notes say the gain "comes from suppressing predictions", and the proportion of valid predictions fell from 0.86 to 0.53.

We also re-tested molmoweb on fresh data, as mentioned in the opening. The final submission's gain of 0.009 on the hidden test shrinks to 0.002 on 3,007 new examples from the same distribution, and the confidence interval rules out 0.009. The change the agent credited most (a new action format) actually hurts a little on the new data.

This is the link between the two kinds of evaluator. A fixed test that gives feedback becomes part of the loop. Then it loses its independence, just like a self-judge.

Co-evolving evaluators: they stop telling right from wrong

MAE is the second example from the opening. Its paper reports a 4.54% average gain across benchmarks. But its reporting pipeline has another problem: the answer extraction script frequently fails to extract the model's actual final answer (in 27–65% of cases). Despite this extraction failure, the external evaluator incorrectly judges 50–88% of these incomplete responses as correct. Once we let the judge see the full answer, none of the released checkpoints is significantly better than the starting model.

MASS uses one model as the worker, the workflow designer, and the evaluator. It reports that its win rate against the starting model rose from 53.9% to 69.9%. MASS released no checkpoints, so we could only audit the paper itself, and three things stand out. First, the "evaluator accuracy" it reports (0.73 to 0.93) measures agreement with outside LLM judges, not with the right answers. Second, on public benchmarks that do have right answers, the model doesn't improve; by the paper's own numbers it gets slightly worse (SWE-bench Verified 62.6 → 61.1, Terminal-Bench 2.0 37.5 → 35.2). Third, in the paper's own appendix, one run rewrote a robotics simulator so the task became trivial, and the outside judges still preferred it 5 times out of 6.

R-Zero trains a model on answers it labels itself. For each generated question, the previous version of the model answers several times, and the majority answer becomes the label. A question is used for training only if the vote is neither too one-sided nor too split. We reproduced two rounds with the authors' code, then had two other models (Qwen3-8B and R1-Distill-Qwen-32B) answer 1,000 of each round's questions. On the questions actually used for training, the labels were right 84.5% of the time in round 1 and only 71.3% in round 2, a drop of 13.2 points (95% CI 6.6 to 20.1). The drop shows up at every level of vote agreement. On top of that, for 67% of round-2 training questions the two reference models couldn't agree on any answer at all, up from 40% in round 1. MATH accuracy still goes up (0.42 → 0.67 → 0.69), but almost all of that comes in round 1. The authors see the same limit: in the paper's own scaling test (Section 5.4), every model size eventually gets worse with more iterations, and the 4B model drops sharply at iteration 4. They doubt that label quality alone explains this. In their recent work, the authors find that the questions themselves also degrade: by round 5, only 39.5% are valid, and among the questions the filter keeps, only 27.64%. This is the same pattern again: the Challenger is rewarded for questions that split the vote, and a question with no clear answer splits the vote, so the loop rewards behavior that serves its signal but not its goal. To tell valid questions from invalid ones, the authors had to use a judge from outside the loop. In either case, the loop's evaluator is the model itself: its errors become the next round's training labels, label quality declines over iterations, and nothing inside the loop can detect this decline.

Hybrid evaluators: the anchor's ceiling stays

EvoLM uses a co-evolving model to write grading rubrics and a frozen 1.7B judge to apply them. The paper shows that a policy trained this way beats policies trained with other reward sources, but it doesn't compare with the model before training, so we did. With the authors' own evaluation code, EvoLM-8B is only about 1 point above the untrained Qwen3-8B on average over 11 tasks, mostly on HumanEval+, where the untrained model's answers are often extracted incorrectly. With the extraction fixed, the gap is 0.4 points and no task improves significantly. None of the other reward sources in the paper's table scores above the untrained model, and the scalar reward model is 8 points below it. So beating the other reward sources is not the same as improving on the starting model. The judge also can't tell correct code from wrong code on HumanEval+: both score around 0.94 (AUC 0.49–0.56).

4What keeps going wrong

Looking across the audits, the same few problems keep coming back. We give each one a short label (D1–D7) for the check that catches it. The appendix explains in a few lines how we computed each check; making them rigorous is our next step.

• The evaluator can't tell good from bad (D1–D3). Given answers whose quality we know, it ranks them at close to chance level (D1), it can't tell good answers from better ones (D2), or it gives almost everything the same score (D3). We saw this in MAE, EvoLM, and AI Scientist v2's reviewer.
• It rewards the wrong thing (D4, D7). The loop finds something the evaluator scores highly but the task doesn't want, usually in a spot the tests don't look at. Think of DGM's shortcuts, molmo2's suppressed predictions, MASS's rewritten simulator, and autoresearch's budget-fitting edits.
• The loop's own score goes up, but an independent check doesn't (D5). MAE, MASS, EvoLM.
• The evaluator drifts as the model trains (D6). R-Zero's labels, and the self-judge in our own simulated loops (Section 5).
• The reported score is picked out of noise. This one isn't about the evaluator itself but about how its score is used. If you run many times on the same test and report the best run, the best looks better than it really is. autoresearch and OpenRSI both do this.

These line up with the dilemma. Fixed evaluators fail by being gamed or by checking too little. Co-evolving evaluators fail by losing their ability to tell good from bad, and by drifting. Hybrids get both problems. And any evaluator whose score feeds back into the loop, even an outside one, can be picked over until its noise looks like progress.

Every one of these problems was found by a check outside the loop. In every case, the loop's own evaluator kept reporting progress. So how much progress we can confirm depends on how far those outside checks reach. Extending that reach is exactly what scalable oversight tries to do, and Sections 7 and 8 come back to it.

5Why it happens: loops where we know the truth

Published systems don't release enough to see the mechanism directly. So we built our own small loops, where we know the true quality of every answer.

Imagine a model iteratively improving its own general ability to answer questions: each round, it writes several answers, an evaluator picks the better ones, and it trains itself on its own winners. A separate reward model, hidden from the loop, tells us how good the answers really are. The only thing we change between loops is the evaluator. Models, data and training settings are in the appendix.

A weaker evaluator means a lower plateau

We took one reward model and made it "blind" above a certain quality level: below that level it scores normally, above it every answer looks the same. Then we moved that ceiling up and down and kept everything else fixed (1.5B model, 3 seeds).

The lower the ceiling, the lower the model ends up. When we raised the ceiling in the middle of a run, progress started again, so the plateau comes from the evaluator, not from the model running out of room.

Three kinds of evaluator, side by side

Next we ran the three kinds of evaluator from Section 2 with Qwen2.5-7B (4 rounds, 2 seeds):

• Fixed: the hidden reward model itself, but capped at the starting model's average, so it can't see anything better than that.
• Co-evolving: the model compares its own answers.
• Hybrid: half of each.

The table checks each evaluator against the hidden reward model. D1: given two answers to the same prompt, how often it picks the better one (0.5 is chance). D2: the same, but only when both answers are above average. D3: the portion of answers that all get the same score. Figures cover rounds 1–4, averaged over 2 seeds.

• The fixed evaluator looks good overall but is blind at the top. It ranks answers best of the three, but once both answers are above its ceiling it is exactly at chance. And as the model improves, the reward distribution becomes increasingly saturated at the ceiling.
• The self-judge is weak from the start and always says things got better. In every round of both seeds, it prefers the new answers over round 1's (58–68% of the time). The true quality doesn't follow: by round 4, one seed is 0.26 better than round 1 and the other is 0.92 worse.
• The hybrid inherits the anchor's blind spot. Above the ceiling only the self-judge half can see, so it does no better there than the self-judge alone.

In short, how much a loop thinks it improved has little to do with how much it really did. The self-judging loop is the most confident, and it's the only one that ends up worse than where it started.

The self-judge also drifts (D6). We had each round's judge score the same fixed set of answers, so any change comes from the judge itself. Its accuracy drops every round, from 0.668 to 0.632 after four rounds. The judge is not biased toward the answers it wrote: judges from other rounds rate those answers about the same. Its ability to tell good answers from bad ones simply declines.

Adding an anchor doesn't fix it by default

"Check the self-judge against an outside anchor" covers many designs: how often the anchor checks in, and how much of the data it grades. We tried both (1.5B model, 2 seeds).

The same basic idea ends anywhere from +0.96 to −1.13 in true quality. When the anchor grades only 10% of the data, every setup ends worse than it started, and the sparsest one is no better than having no anchor at all. Across these ten setups, the more a loop believed it improved, the worse it tended to actually do (r = −0.58, p = 0.08).

These are the same failures we found in the audits: weak grading (D1, like MAE's and EvoLM's judges), progress the truth doesn't confirm (D5, like MAE and MASS), and drift (D6, like R-Zero). They also add one the audits couldn't test, because no system released the per-round data we'd need: blindness at the top (D2). Here we can check all of them against known answers, which also shows our checks measure what they're supposed to.

6What an audit should check

Our main point is simple: a self-improvement claim should count only if something that didn't steer the loop confirms it, and only as far as that check reaches. The steps below are what we did by hand for each system. They're a first sketch, not a finished protocol, and making each check rigorous and general is what we're working on next.

No audit can solve the dilemma. If an improvement is beyond what any outside check can see, no audit can confirm it. What an audit can do is make that limit visible, so it can't be ignored:

• the check used to confirm a gain is one the loop never optimized against;
• every claim states how large the gain is, what the independent check actually measured, and how uncertain the result is;
• if the loop's own evaluator has stopped working, the audit should catch it, and hence the loop's self-reported gains should be disregarded.

The four steps:

1. List every evaluator in the loop. What does it score? Does it share weights with the model? Can the model see its score? How strongly does the loop optimize for its score? This tells you which signals can still confirm anything, and which checks to run.
2. Test each evaluator on examples whose quality you already know (D1–D7). What you can test depends on what's released: final weights are enough for D1, D3, D4 and D7; per-round checkpoints or logs let you run all seven.
3. Re-check the reported numbers. Use the same evaluation code for start and end, report paired confidence intervals, check the grader against the raw outputs, and re-test the best run on fresh data.
4. Give a verdict. For example: holds; holds but overstated; doesn't reproduce; or the evaluator itself fails.

We packaged these steps into a toolkit, rsi-audit (release pending). We plan to open-source it together with a registry that the community can add to: one card per RSI system, saying what it claims, which signal confirms it, which checks have been run, and the verdict. It can also work as a leaderboard, but one that ranks systems by how much of their claimed improvement holds up, not by how much they claim.

Auditing a live loop

The same checks can also run before a loop publishes an update, not just after the paper is out. Reef, a recent open framework for self-improving agents, by default publishes whatever version the loop's own judge scores highest. Take the simplest example: on a prompt-optimization task (GEPA, Qwen2.5-7B, MATH-500), that default published prompts the judge rated close to 1.0 while strict accuracy dropped to 0. A gate that checks a held-out set with known answers blocked those versions and published the real improvement. But it could only confirm what its own held-out set measures, which is the ceiling side of the dilemma again.

7Ways out

None of this means useful self-improvement is impossible. A loop stays honest as long as information from outside keeps coming in. There are three main ways that can happen, and our audits show the limits of each.

Generation–verification asymmetry

A weaker evaluator can supervise a stronger model when checking an answer is much easier than producing it. That's the idea behind prover–verifier games, debate, critique, and other scalable-oversight methods: make the task easy for the evaluator to check, instead of making the evaluator as strong as the model.

When the score is exactly the goal, a fixed evaluator really can confirm progress. CORAL's released Erdős construction and its 1,103-cycle kernel both held up when we checked them ourselves, because a math construction is its own proof, and a cycle count on a frozen simulator is the result.

But this way out has at least three limits. First, coverage: hand-built checks have gaps. Running DGM's tests is much cheaper than writing a fix, but 3 of 10 new passes still slipped through gaps in the tests. CORAL's agents didn't cheat, but its graders can be fooled too: they accepted two fake submissions we wrote, one claiming an Erdős bound 1e-4 better than its construction achieves, and one rewriting the simulator's cycle counter to claim 900 cycles. Both holes come from standard graders that CORAL reuses: OpenEvolve's Erdős evaluator and the test harness of Anthropic's kernel take-home. Second, many tasks don't have this asymmetry. Choosing a research direction or judging a new hypothesis is not easier to check than to do. Third, checking gets harder as outputs get longer and more complex, so the advantage can shrink as models get stronger.

The real world as evaluator

Running code, experiments, simulations and physical outcomes bring in evidence from outside the loop. But this only works when the results can be observed, arrive in time, and are hard to manipulate, and only as long as the loop isn't tuning on them. For example, autoresearch's validation loss and OpenRSI's hidden tests are real outside signals, but once they steer the loop, the best reported score partly reflects noise rather than real progress. In MASS, the agent simply rewrote the simulator.

Consistency checks

Self-consistency, majority voting and logical constraints can catch some errors without any labels. But being consistent is not the same as being right. In our R-Zero reproduction, the accuracy of the majority-vote training labels fell from 0.85 to 0.71 in a single round, and a loop can converge on a standard that is self-consistent but wrong.

8Scalable oversight comes first

Push the dilemma to its end and it points to a real limit on open-ended RSI. Picture a closed loop working on tasks beyond any outside judge. It writes its own tasks, trains its own solver, updates its own evaluator, and decides for itself whether each new version is better. If no outside information comes in, a rising internal score can't tell real progress from the loop drifting deeper into its own mistakes.

That doesn't make all self-improvement impossible. Within the reach of an independent check, gains can be confirmed, and our audits found some: DGM really did go from 2 to 9 fixed tasks, and autoresearch's edits hold up at 30 minutes and on new text. The hard part is going beyond that reach. Open-ended self-improvement that can be confirmed needs a source of ground truth that remains reliable as the model gets stronger. Where verification is easier than generation, scalable oversight can extend the evaluator's reach. Where the world gives observable feedback, the loop can get new evidence. Where neither holds, self-improvement may be impossible to verify, even in principle. And even today, the evaluators these systems rely on cover less than the claims they support: unit tests for "a better coding agent", a 5-minute loss for "a better model", 72 videos for a research task, and three human reviews for "passed peer review".

We usually think of scalable oversight as a safety tool for overseeing future AI systems that outperform humans. For recursive self-improvement, it may play a more basic role. Scalable oversight may not just be a safety add-on to RSI. It may also be what makes continued self-improvement possible to steer and to measure at all; this is what we call scalable supervision. And that may be one of the fundamental limits of open-ended recursive self-improvement.

What would help right now:

• Release per-round checkpoints and evaluator logs, so others can measure drift and check whether the loop's score reflects actual performance.
• Report the quality of the loop's own evaluator: how well it tells good from bad on examples with known answers, and whether its scores saturate.
• Keep one test the loop never sees, and report how many times any visible test was used to select among versions.

Appendix: how we measured

References

@misc{liu2026evaluatordilemma,
  title  = {The Evaluator Dilemma: Does Recursive Self-Improvement Really Improve?},
  author = {Liu, Yanchen},
  year   = {2026},
  note   = {Blog post}
}