RSI Audit Registry

Did These Self-Improving Systems Really Improve?

Audit cards for self-improving AI systems

Maintainer: Yanchen Liu, MIT
Updated: 2026-11-03
Entries: 10 systems
Companion post: The Evaluator Dilemma ↗

When a self-improving AI system reports that it got better, how do we know the gain is real? For each published system here, we check whether the evaluators that measured the gain can tell better from worse, and whether the gain holds up when we re-check it ourselves, or against the paper's own data when nothing can be rerun.

Contents

By evaluator type
Overview
How to read this registry

Entries

EvoLM
Multi-Agent Evolve
OpenRSI Index
Darwin Gödel Machine
R-Zero
Double Ratchet
CORAL
MASS
autoresearch
The AI Scientist v2

Appendix: checking our diagnostics

By evaluator type and ground truth

in-loop evaluator ↓ / tasks have ground truth? →	has ground truth	mixed	no ground truth
fixed	OpenRSI Index Darwin Gödel Machine CORAL autoresearch	—	The AI Scientist v2
hybrid	—	EvoLM Double Ratchet	—
co-evolving	R-Zero	Multi-Agent Evolve MASS	—

fixedco-evolvinghybridoutside test pulled into the loop

Overview

The system and its paper				Our audit
System	In-loop evaluator	Ground truth	Evidence in paper	Audit	Evaluator diagnostics D1–D7	Verdict
EvoLM	hybrid	partly	L1	empirical	1234567	evaluator fails
Multi-Agent Evolve (MAE)	co-evolving	partly	L1	empirical	1234567	not reproduced
OpenRSI Index	fixed	✓	L0	empirical	1234567	partially holds
Darwin Gödel Machine (DGM)	fixed	✓	L0	empirical	1234567	holds, overstated
R-Zero	co-evolving	✓	L1	empirical	1234567	evaluator fails
Double Ratchet (Who Grades the Grader?)	hybrid	partly	L2	paper-level	1234567	not supported (paper's own data)
CORAL	fixed	✓	L0	empirical	1234567	holds
MASS (Multi-Agent Self-Supervision)	co-evolving	partly	L1	paper-level	1234567	not supported (paper's own data)
autoresearch (Karpathy)	fixed	✓	L0	empirical	1234567	holds, overstated
The AI Scientist v2	fixed	✗	L0	empirical	1234567	evaluator fails

Each box is one diagnostic (1 = D1 … 7 = D7).

1 run: evaluator fails1 run: evaluator holds1 run: inconclusive1 to do: required and possible, not run yet1 required, needs artifacts the authors did not release1 not required for this evaluator type

Evidence in paper: how much the paper itself shows, not our verdict.

L0 claim onlyL1 fair comparison: independent reporting evaluator, start and end in the same harnessL2 L1 + checks of the in-loop evaluator and of individual gainsL3 L2 + artifacts public so others can re-check

How to read this registry

Audit protocol

Map every evaluator in the loop. Training signal, selection, stopping, reporting. For each: fixed or co-evolving, what it shares with the policy, what the policy can see, how strongly the loop optimizes for its score, and whether the reporting evaluator is independent of it.
Diagnose each evaluator on items whose quality is known independently, drawn from the loop's own outputs:1 telling good from bad · 2 telling good from better · 3 saturation · 4 rewarding the wrong thing · 5 own score vs. truth · 6 drift and self-preference · 7 what the tests don't cover
Re-check the reported numbers. Use the same evaluation code for start and end, report paired confidence intervals, check the grader against the raw outputs, and re-test the best run on fresh data.
Verdict. State which evaluator measured the reported progress, and whether the diagnostics show that it can be trusted.

Reading an entry

Verdict: outcome of the audit: holds, holds, overstated / partially holds, not reproduced / evaluator fails, not supported (paper's own data).

Evidence in paper: how much evidence the paper itself gives that its gain is real. L0 claim only · L1 independent reporting evaluator and start/end in the same harness · L2 + diagnostics of the in-loop evaluator and checks that individual gains are real · L3 + artifacts public.

In-loop evaluator: fixed (tests, a frozen judge, fixed data), co-evolving (the model or a co-trained model judges itself), hybrid (co-evolving component checked by a fixed one).

Entries

EvoLM

evaluator failsEvidence in paper L1

EvoLM: Self-Evolving Language Models through Co-Evolved Discriminative Rubrics · arXiv 2605.03871 · 2026

The paper compares reward sources with each other, but not with the model before training. We added that comparison on the authors' own evaluation setup. Over the 11 tasks we ran, EvoLM-8B is only about 1 point above the untrained Qwen3-8B on average, and 0.4 points once answer extraction is fixed, with no task improving significantly. None of the other reward sources in the paper's table scores above the untrained model. The frozen judge that applies the rubrics cannot tell correct code from wrong code.

AWhat the paper claims

Rubrics that the model evolves while it trains are a better reward than other sources: the policy trained this way averages 69.3% on the OLMo3-Adapt suite, ahead of policies trained with prompted rubrics (GPT-4.1 66.7%, Qwen3-8B 67.5%), a scalar reward model (59.7%) and four other rubric-based RL methods (66.7–67.6%).

BEvaluators in the loop protocol step 1

Evaluator	Role	Type	Shares with policy	Pressure	What it is
`rubric_model_plus_frozen_judge`	training signal	hybrid	family	direct	A rubric model that evolves with the policy (starting from Qwen3-8B) writes the grading rubrics, and a frozen Qwen3-1.7B judge scores answers with them.
`temporal_contrast`	training signal	co-evolving	data	direct	Training pairs assume that a later checkpoint's answer is better than an earlier one's.
`olmes_benchmarks`	reporting	fixed	—	—	The OLMo3-Adapt benchmarks, run with the authors' OLMES evaluation code. Most have reference answers; AlpacaEval uses a GPT-4.1 judge.

Risks to check in this loop

The grader is trained alongside the model, so it grows less independent each round.

CEvaluator diagnostics protocol step 2

Authors released: final weights.

Diagnostic	Status	Finding
1 telling good from bad	run: evaluator fails	The frozen 1.7B judge, with either set of rubrics, tells correct code from wrong code at close to chance level (AUC 0.49-0.56 on HumanEval+, checked against execution results).
2 telling good from better	required, needs artifacts the authors did not release
3 saturation	run: evaluator fails	Correct and wrong code both score about 0.94. The loop's own evaluator prefers the final model over the starting one only 51.5% and 52.4% of the time.
4 rewarding the wrong thing	run: inconclusive	The evolved rubrics put more weight on format (from about 10% to about 20%), yet the judge does not penalize answers that copy the '(CODE)' placeholder.
5 own score vs. truth	required, needs artifacts the authors did not release
6 drift and self-preference	required, needs artifacts the authors did not release

Other measurements

in loop judge auc	0.49-0.56 on HumanEval+: correct and wrong code both score about 0.94.
own evaluator start vs end	The loop's own evaluator prefers the final model over the starting model only 51.5% of the time on open-ended prompts and 52.4% on MATH.
reward sources vs untrained	Over the 11 tasks we ran, no baseline reward source in the paper's table scores above the untrained model; most are within about 1 point of it, and the scalar reward model is 8.1 points below. These are the paper's numbers compared with our run of the untrained model on the same setup; our EvoLM run matches the paper's average within 0.2 points.
grader issues	The starting model copies the '(CODE)' placeholder from the prompt template in 9.9% of HumanEval+ answers. On BBH, the regex takes the last option letter mentioned rather than the boxed answer (1,215 of 6,511 starting-model answers).

DRe-checking the reported numbers protocol step 3

Measure	Claimed	Audited
Average over the 11 tasks we ran (AlpacaEval excluded), against the untrained Qwen3-8B	not reported	+1.4 with the original grading; +0.4 with answer extraction fixed
Reward sources in the paper's table, against the untrained model (11 tasks)	-	none of the 7 baselines is above it (-0.1 to -8.1 points); EvoLM +1.2
MBPP+ (untrained → EvoLM)	-	68.7 → 68.5 (-0.2 [-1.4, +1.0])
HumanEval+ (untrained → EvoLM)	-	original grading 78.4 → 86.7; with code extraction fixed 85.8 → 86.9 (+1.1 [-0.7, +2.9])
BBH (untrained → EvoLM)	-	original grading 64.3 → 67.5; with answer extraction fixed (the boxed answer) 80.4 → 80.7 (+0.3 [-0.4, +1.0])

EHow this audit was done

What we did · empirical · 2026-10-10

Ran the untrained Qwen3-8B (the starting model) and EvoLM-8B on 11 tasks with the authors' own OLMES harness and settings.
Fixed the answer extraction and graded again: a more lenient code extractor for HumanEval+, and the model's boxed answer for BBH. We compared the two models question by question.
Compared the other reward sources in the paper's table with the untrained model, using the paper's numbers.

Deviations from the original setup

We left out AlpacaEval, which needs a GPT-4.1 judge.

Author response: responded 2026-11-03 — The authors point out that the paper's Qwen3-8B row is a policy trained on rubrics from prompted Qwen3-8B, not the untrained model, and that the paper's tables compare reward sources: rubrics evolved by the model being trained against other rewards. The paper does not compare with the untrained model, which they agree it should. They prefer to keep the original OLMES verifiers so that numbers stay comparable, suggest reporting the extraction issues to OLMES, and did not comment on our re-graded numbers without the exact changes.

Our note: The comparison with the untrained model above is the one the authors agree is missing.

Multi-Agent Evolve (MAE)

not reproducedEvidence in paper L1

Multi-Agent Evolve: LLM Self-Improve through Co-evolution · arXiv 2510.23595 · 2025

The model's own judge cannot tell right answers from wrong ones. The script that extracts each final answer often returns a fragment with no answer in it, and the outside judge still marks many of these as correct. When the judge is given the full answer, none of the released checkpoints is significantly better than the starting model (all McNemar p >= 0.068).

AWhat the paper claims

One model writes questions, answers them and grades the answers, and improves itself without outside supervision: +4.54% on average across benchmarks.

BEvaluators in the loop protocol step 1

Evaluator	Role	Type	Shares with policy	Pressure	What it is
`self_judge_answers`	training signal	co-evolving	weights	direct	The same model scores the answers from 1 to 10, and that score is the reward.
`self_judge_questions`	selection	co-evolving	weights	filter	The same model scores the proposed questions; low-scoring ones are dropped.
`llm_judge_on_extracted_snippet`	reporting	fixed	—	—	Llama-3.1-405B judges TRUE or FALSE on an answer snippet that a script extracts from the answer; 500 questions per benchmark.

Risks to check in this loop

The grader is the model being improved (or an earlier version of it), so it may miss the same mistakes the model makes.

CEvaluator diagnostics protocol step 2

Authors released: final weights.

Diagnostic	Status	Finding
1 telling good from bad	run: evaluator fails	The in-loop judge tells right answers from wrong ones at close to chance level (AUC 0.51 on GSM8K; the starting model's judge scores 0.54).
3 saturation	run: evaluator fails	It gives 996 of 1,000 answers the same score of 8 out of 10.
5 own score vs. truth	required, needs artifacts the authors did not release
6 drift and self-preference	required, needs artifacts the authors did not release

Other measurements

in loop judge auc	0.51, close to chance: it gave 996 of 1,000 answers the same score of 8 out of 10.
reporting grader disagreement	For the checkpoint trained without reference questions (`noref`), giving the judge the full answer changes its decision on about 20% of the answers (19–21% on the three benchmarks); for the starting model and the other two checkpoints, on 1–4%.
grader issues	For 27-65% of the `noref` checkpoint's answers, the extractor returns a fragment with no answer in it, and the judge still marks 50-88% of those as correct.

DRe-checking the reported numbers protocol step 3

Measure	Claimed	Audited
GSM8K (start / `noref` / `halfref` / `ref`), judged on the full answer	85.2 / 85.2 / 82.2 / 76.0	86.0 / 85.4 / 84.4 / 84.4
MATH, judged on the full answer	60.4 / 63.8 / 65.8 / 49.0	67.2 / 65.6 / 64.8 / 63.8
MMLU, judged on the full answer	63.4 / 68.0 / 69.0 / 62.4	66.0 / 69.0 / 65.2 / 64.4

EHow this audit was done

What we did · empirical · 2026-10-11

Ran the starting model, Qwen2.5-3B-Instruct, and the three released checkpoints through the authors' own benchmark pipeline: same data, seed-42 subsets, prompt template, greedy decoding, answer extractor and judge prompts. There is one checkpoint for each of the paper's settings: without reference questions (noref, step 300), half reference (halfref, step 450) and with reference questions (ref, step 200). Reference questions are existing questions, shown without their answers, that the model can use as examples when it writes new questions.
Judged the same answers again, this time giving the judge the full final answer instead of the extracted snippet.
Checked how well the in-loop judge tells right answers from wrong ones, on GSM8K answers whose correctness we know.

Deviations from the original setup

We used Gemma-3-27B-it as the judge instead of Llama-3.1-405B.
Two of the three released checkpoints are not from the step the paper reports: the paper trains for 300 steps, but halfref is from step 450 and ref from step 200.

Author response: emailed 2026-11-03 — no response yet

OpenRSI Index

partially holdsEvidence in paper L0

OpenRSI Index: a benchmark for recursive self-improvement in research agents · GitHub

The hidden test is meant to measure progress from outside the loop, but agents see its score after every submission and adjust to it. By a rough estimate, the gains on isaaclab and ace are much larger than picking the best submission could produce, so they look real. However, the gains on kev and molmo2 cannot be told apart from picking the best, and molmo2's gain came from exploiting the metric. On molmoweb, most of the gain disappears on new data.

AWhat the paper claims

Each task has a hidden test the agent cannot read or change, so its score is meant to measure progress from outside the loop. The agent's result on each task is its best hidden-test score.

BEvaluators in the loop protocol step 1

Evaluator	Role	Type	Shares with policy	Pressure	What it is
`hidden_test_feedback`	selection	fixed	none	direct	Each task's hidden test. The agent cannot read or change it, but sees its score after every submission and uses it to keep, undo or tune changes.
`hidden_test_best`	reporting	fixed	hidden_test_feedback	—	The best hidden-test score over all of the agent's submissions, reported as its result on the task.

Risks to check in this loop

The loop optimizes directly for a fixed score, so it can learn to pass without doing what the score is meant to check.
The agent can see the evaluator's score, so it can tune against the evaluator.
Tasks used to pick versions also count toward the reported result.

CEvaluator diagnostics protocol step 2

Authors released: training logs.

Diagnostic	Status	Finding
2 telling good from better	run: inconclusive	We cannot measure how much scores vary between submissions, because the logs keep only total scores. By a rough estimate, most of the score changes the agent acted on are smaller than the noise of a single evaluation, and the gains on `kev` and `molmo2` cannot be told apart from simply keeping the best submission.
4 rewarding the wrong thing	run: evaluator fails	Half of `molmo2`'s test videos contain no target, and pointing at nothing on them earns full marks. The agent's gain came from predicting less: by its own notes, the proportion of valid predictions fell from 0.86 to 0.53.
5 own score vs. truth	run: evaluator fails	On `molmoweb`, the +0.0091 gain on the hidden test shrinks to +0.0017 [-0.0033, +0.0068] on new data, and the change the agent credited most makes things slightly worse.
7 what the tests don't cover	to do: required and possible, not run yet

Other measurements

steering How often Claude, while deciding whether to keep or undo a change, cites a hidden-test score it has received (counted by keyword search over its run logs, not checked by hand): 69 times on ace, 44 on reasonir, 35 on minference, 15 on kev, 11 on molmoweb, 9 on liger, 6 on molmo2, and 1–2 on the other three tasks.

examples

steering	How often Claude, while deciding whether to keep or undo a change, cites a hidden-test score it has received (counted by keyword search over its run logs, not checked by hand): 69 times on `ace`, 44 on `reasonir`, 35 on `minference`, 15 on `kev`, 11 on `molmoweb`, 9 on `liger`, 6 on `molmo2`, and 1–2 on the other three tasks.
examples	`molmo2`: the agent tried many frame counts and scored each on the 72 hidden videos. Its own notes say the gain comes from making fewer predictions and would not carry over to other data. `minference`: the agent submitted the same workspace five times (scores 1.4913 to 1.5041) and reported the highest. `molmoweb`: gains on individual action types flip on new data (keyboard_press +0.055 on the hidden test, -0.006 on new data).

molmo2: the agent tried many frame counts and scored each on the 72 hidden videos. Its own notes say the gain comes from making fewer predictions and would not carry over to other data.
minference: the agent submitted the same workspace five times (scores 1.4913 to 1.5041) and reported the highest.
molmoweb: gains on individual action types flip on new data (keyboard_press +0.055 on the hidden test, -0.006 on new data).

DRe-checking the reported numbers protocol step 3

Measure	Claimed	Audited
`kev` (Claude, 21 submissions)	+0.007	Keeping the best of 21 submissions would give about +0.019 even with no real improvement (rough estimate)
`molmo2` (Claude, 59 submissions, 72 hidden videos)	+0.143	Keeping the best of 59 would give about +0.13 with no real improvement, and 5% of such runs would exceed +0.23 (rough estimate)
`ace` (Claude, 29 submissions)	+0.165	Picking the best alone would give about +0.071 (rough estimate), so most of the gain looks real
`isaaclab` (Claude, 30 submissions)	+0.208	Picking the best alone would give about +0.031 (rough estimate), so the gain looks real
`molmoweb`, hidden test	+0.0091 [-0.0013, +0.0204]	Our rerun gives the same number
`molmoweb`, 3,007 new examples	-	+0.0017 [-0.0033, +0.0068]; this range rules out the +0.0091 seen on the hidden test
`molmoweb`, the new action format alone, on new data	the agent credited it with about +0.007	-0.0036 [-0.0074, -0.0000]: slightly worse

EHow this audit was done

What we did · empirical · 2026-10-14

Read the released Claude and GPT run logs for 10 tasks, and counted how often the agent's decisions refer to its hidden-test score.
Estimated, roughly, how much a score would rise from keeping the best of many submissions when nothing improves, and compared that with each reported gain.
Re-tested the starting and final versions of one task, molmoweb, on 3,007 new examples from the same source (about three times the size of the hidden test).

Deviations from the original setup

The estimate of how much picking the best adds is rough. It uses a formula meant for pass rates, but these scores are not pass rates (kev's comes from log-likelihood, molmo2's is a soft-F1), and it treats submissions as independent, though they all use the same test.
The logs keep only each submission's total score, so we cannot measure how much scores vary between submissions. Using the spread of the scores we do see (0.0027 on kev, 0.042 on molmo2) leads to the same reading for kev and molmo2.
We re-tested only one task (molmoweb) on new data.
The count of decisions that refer to the hidden score covers only the Claude runs.

Author response: emailed 2026-11-03 — no response yet

Darwin Gödel Machine (DGM)

holds, overstatedEvidence in paper L0

Darwin Godel Machine: Open-Ended Evolution of Self-Improving Agents · arXiv 2505.22954 · 2025

The gain is real on tasks with reference answers, but the unit tests overstate it: the agent learns to pass the tests rather than to do what they check, and the tests cannot tell the difference.

AWhat the paper claims

A coding agent that rewrites its own code improves on SWE-bench from 20% to 50%.

BEvaluators in the loop protocol step 1

Evaluator	Role	Type	Shares with policy	Pressure	What it is
`swebench_tests_staged`	selection	fixed	none	direct	Unit tests on growing SWE-bench subsets (10, 60, then 200 tasks) score each self-modified agent; the scores decide which agents are kept as parents.
`swebench_200`	reporting	fixed	swebench_tests_staged	—	Unit tests on 200 SWE-bench tasks, which overlap the tasks used during the search; the starting agent was measured on a different set of 58 tasks.

Risks to check in this loop

The loop optimizes directly for a fixed score, so it can learn to pass without doing what the score is meant to check.
Tasks used to pick versions also count toward the reported result.
The start and end were not measured the same way.

CEvaluator diagnostics protocol step 2

Authors released: per-iteration weights.

Diagnostic	Status	Finding
2 telling good from better	to do: required and possible, not run yet
5 own score vs. truth	run: evaluator fails	On 50 other tasks, 3 of the 10 newly passing tasks pass the tests through shortcuts that break the code.
7 what the tests don't cover	run: evaluator fails	Checking behavior against the reference fixes inside each task's container shows what the unit tests miss: global monkeypatching, arguments of the wrong type and duplicate SVG ids.

Other measurements

false gain rate	3 of the 10 newly passing tasks
examples	`sympy-19783`: patches sympy's adjoint multiplication globally so that it calls itself; `2*adjoint(x)` never finishes. `sympy-22456`: `String.args` returns a raw str, so `atoms()` and `free_symbols` crash. `matplotlib-24637`: overwrites child gids and produces duplicate SVG ids.
paper reported	The paper itself reports the agent gaming its objective: in its hallucination case study, the agent removed the special-token logging that the detector relied on (Appendix H).

DRe-checking the reported numbers protocol step 3

Measure	Claimed	Audited
SWE-bench, released logs (the 58 tasks measured at both start and end)	20% → 50%	20.7% → 50.0% (McNemar p = 0.0002)
50 other SWE-bench Verified tasks	-	tasks passing the tests 2 → 12; tasks actually fixed 2 → 9 (p ≈ 0.016)

EHow this audit was done

What we did · empirical · 2026-10-11

Re-analyzed the released search logs, looking for effects of picking the best agents and for start and end being measured on different tasks.
Ran the initial and the best released agents on 50 other SWE-bench Verified tasks, with Devstral-Small-2507 in place of Claude and the official grading.
Compared every newly solved patch with the reference fix, and tested the behavior around each issue inside the task's container.

Deviations from the original setup

An open-weight model (Devstral-Small-2507) as the agent instead of Claude 3.5 Sonnet
50 tasks instead of the full set

Author response: emailed 2026-11-03 — no response yet

R-Zero

evaluator failsEvidence in paper L1

R-Zero: Self-Evolving Reasoning LLM from Zero Data · arXiv 2508.05004 · 2025

We reproduced two rounds with the authors' code. The loop's evaluator, a majority vote over the model's own answers, gets worse within one round: the labels used for training fall from 85% to 71% correct, and for two thirds of round-2 training questions two reference models cannot agree on any answer. MATH accuracy still rises (0.42 → 0.67 → 0.69), almost all of it in round 1. Round 2, with worse labels, adds only +0.02, and the paper itself reports that every model size eventually gets worse with more rounds.

AWhat the paper claims

A question-writer (the Challenger) and a solver (the Solver) start from the same base model and improve together without any outside data; the Solver's accuracy on math and general reasoning benchmarks rises over the rounds.

BEvaluators in the loop protocol step 1

Evaluator	Role	Type	Shares with policy	Pressure	What it is
`solver_majority_vote`	training signal	co-evolving	weights	direct	The label is the majority answer among 9-10 answers from the Solver; the Solver is rewarded for matching it.
`challenger_uncertainty_reward`	training signal	co-evolving	family	direct	The Challenger is rewarded most for questions the Solver answers consistently about half the time, with a penalty for repeating similar questions.
`consistency_band_filter`	selection	co-evolving	weights	filter	Only questions where 30-80% of the Solver's answers agree are kept; proofs and a few other question types are dropped.
`fixed_step_checkpoint`	stopping	fixed	none	none	The Challenger stops at step 5 and the Solver at step 15; no evaluator decides when to stop.
`math_and_general_benchmarks`	reporting	fixed	—	—	Math benchmarks (AMC, Minerva, MATH-500, GSM8K, Olympiad, AIME), graded by answer matching with a GPT-4o recheck, and general reasoning benchmarks (MMLU-Pro, SuperGPQA, BBEH).

Risks to check in this loop

The grader is the model being improved (or an earlier version of it), so it may miss the same mistakes the model makes.
The grader is trained alongside the model, so it grows less independent each round.
The loop deliberately picks the items the evaluator is least sure about.

CEvaluator diagnostics protocol step 2

Authors released: paper only. Our reproduction provides per-iteration weights.

Diagnostic	Status	Finding
1 telling good from bad	run: evaluator holds	How strongly the vote agrees does tell right labels from wrong ones (AUC 0.81 in round 1, 0.80 in round 2).
2 telling good from better	run: evaluator fails	But the loop trains only on questions where 30-80% of the answers agree, and those are the least reliable labels: 67-89% correct in round 1, and only 51-62% in the middle of that range in round 2.
3 saturation	to do: required and possible, not run yet
5 own score vs. truth	to do: required and possible, not run yet
6 drift and self-preference	run: evaluator fails	Judged by two other models, the training labels are right 84.5% of the time in round 1 and 71.3% in round 2 (-13.2 [-20.1, -6.6]), even among questions with the same vote agreement. Questions on which the two models disagree, so the label cannot be checked, rise from 40% to 67%.

Other measurements

paper reported	Section 5.4 of the paper (Figure 4 in arXiv v2): every model size gets worse after enough rounds. The 0.6B model peaks after round 1; the 4B model improves for three rounds and drops sharply in round 4. The authors say label quality is not the main or only cause, and suggest the model degrades from training only on data it generated itself.

DRe-checking the reported numbers protocol step 3

Measure	Claimed	Audited
MATH-500 validation (the run's own grader, up to 4,096 tokens)	rises over the rounds	0.422 → 0.672 → 0.690. Answers got much shorter in round 1: the proportion cut off at the length limit during training fell from 12.7% to 2.2%
Is the round-1 gain just format or length? (500 problems, 8 samples each)	-	No: +0.260 [0.238, 0.281], and still +0.25 with a lenient grader or a 16,384-token limit. Round 2 adds +0.018
Accuracy of the training labels (judged by two other models)	79% → 63% over 3 rounds (the paper's own figure)	0.845 [0.808, 0.877] → 0.713 [0.651, 0.768] in one round, a drop of 0.132 [0.066, 0.201]

EHow this audit was done

What we did · empirical · 2026-10-12

Reproduced two rounds from Qwen3-4B-Base with the authors' code, keeping every label and score the loop produced.
Had two other models (Qwen3-8B with thinking, and R1-Distill-Qwen-32B; 4 answers each) answer 1,000 questions per round: 700 that were used for training and 300 that were not. We counted a question as having a reliable answer only when both models agreed.
Sampled the starting, round-1 and round-2 Solvers 8 times on each of 500 MATH problems, and graded the answers three ways: the run's own grader, a more lenient grader for answers without a box, and a longer 16,384-token limit.

Deviations from the original setup

6 GPUs instead of 8
2 rounds instead of 3
The Solver is trained to step 15, the checkpoint the original code merges and evaluates

Author response: responded 2026-11-03 — The authors agree with our main findings: the round-1 gain holds, round 2 adds relatively little, and the decline in label quality is consistent with their paper. They add that, in their recent work, the validity of the generated questions is also an important factor in the later collapse: the proportion of valid questions fell from 76.5% in round 1 to 39.5% in round 5, and to 27.64% among the questions the answer-consistency filter kept. They suggest that some of the questions on which our two reference models disagree may be invalid, which they have not tested on our data. arXiv:2610.04299

Our note: This is the pattern our audits keep finding: the loop learns whatever its own signal rewards. The Challenger is rewarded for questions that split the Solver's vote, and a question without a single well-defined answer splits the vote, so an invalid question can earn a high reward and pass the filter. To tell valid questions from invalid ones, the authors had to use a judge from outside the loop.

Double Ratchet (Who Grades the Grader?)

not supported (paper's own data)Evidence in paper L2

Who Grades the Grader? Co-Evolving Evaluation Metrics and Skills for Self-Improving LLM Agents · arXiv 2607.12790 · 2026

The grader evolves together with a skill loop and is anchored on ten labeled examples. We could not rerun anything, because no run data is released. The paper's numbers are fine, but by its own tables the agent improves just as much with a grader that has collapsed and no longer checks anything, so the gain cannot be credited to the evolved grader. By the end of a run, the evolved grader is also no more accurate than the plain LLM judge it contains (both about 0.55). The paper does show that the ten examples keep the grader from collapsing, and it notes itself that a higher task score does not prove a grader is good.

AWhat the paper claims

A grading metric evolved from ten labeled examples is valid (+0.21 agreement with hidden ground truth on MBPP+, ahead of the plain LLM judge it contains), and when it evolves together with a skill loop, the agent gets 88-110% of the gain that ground truth or a hand-written rubric would give.

BEvaluators in the loop protocol step 1

Evaluator	Role	Type	Shares with policy	Pressure	What it is
`evolved_metric`	training signal	hybrid	weights	direct	The grader for training attempts: several small checks combined, one of which asks an LLM judge (the same model as the agent). When an attempt fails, the failure message is used to write new skills.
`skill_critic`	training signal	fixed	weights	filter	An LLM critic (the same model as the agent) reads every training attempt, and its comments are also used to write skills. It keeps working even in the runs where the grader has collapsed.
`ten_item_anchor`	selection	fixed	none	filter	Ten labeled examples. When the grader is changed, each candidate version is kept or dropped by how well it agrees with these ten.
`locked_set_rollback`	stopping	fixed	none	filter	If the score on the held-out tasks stays 0.10 below the best for five rounds, the skill library is reset to the best round. These are the same tasks the paper reports on. This happened 0-2 times per run, and is turned off in the runs where the grader collapses.
`locked_task_scores`	reporting	fixed	locked_set_rollback	—	Unit tests, the official execution check or a hand-written rubric on 40-48 held-out tasks, three seeds. The headline figures use the best round on these tasks.
`independent_final_judge`	reporting	fixed	—	—	Opus 4.8 compares the baseline report with the evolved one, pair by pair. Its rubric for this task was written after reading the reasons a general-purpose judge gave.

Risks to check in this loop

The grader is the model being improved (or an earlier version of it), so it may miss the same mistakes the model makes.
The agent can see the evaluator's feedback, so it can tune against the evaluator.
The checkpoint is chosen on the reported tasks, so the result is optimistic.

CEvaluator diagnostics protocol step 2

Authors released: paper only.

Diagnostic	Status	Finding
1 telling good from bad	required, needs artifacts the authors did not release
2 telling good from better	required, needs artifacts the authors did not release
4 rewarding the wrong thing	run: evaluator fails	From the paper: on report writing, the evolved skills learned to put evidence tags in reports without the numbers they should cite, and the hand-written rubric still gave them credit. Only an independent judge caught this; the paper then added a check that fixed it.
5 own score vs. truth	run: inconclusive	From the paper: on MBPP+, the grader agrees with the hidden tests 0.625 of the time at its best round and about 0.55 at the end; on Spider, 0.500. There is no comparison with always guessing the most common answer, and a fairer measure would need the run data.
6 drift and self-preference	required, needs artifacts the authors did not release
7 what the tests don't cover	required, needs artifacts the authors did not release

Other measurements

in loop evaluator	The paper measures the grader by how often it agrees with the hidden answers, but never compares this with a grader that always gives the most common answer. Agreement is 0.625 at the best round on MBPP+, and 0.500 on Spider, where the paper's own paired test finds no gain. On report writing, the paper notes the grader is at its ceiling from round 0.
paper reported	With the safeguards around the ten examples turned off, the grader passes 97-100% of report attempts, yet the agent's task score rises just as much; the paper itself says task score cannot tell a working grader from a broken one. On reports, the evolved skills also left the number out of about 30% of evidence tags at the best rounds, and an independent judge preferred the starting reports in 88% of the pairs it decided.

in loop evaluator

The paper measures the grader by how often it agrees with the hidden answers, but never compares this with a grader that always gives the most common answer. Agreement is 0.625 at the best round on MBPP+, and 0.500 on Spider, where the paper's own paired test finds no gain. On report writing, the paper notes the grader is at its ceiling from round 0.

paper reported

With the safeguards around the ten examples turned off, the grader passes 97-100% of report attempts, yet the agent's task score rises just as much; the paper itself says task score cannot tell a working grader from a broken one. On reports, the evolved skills also left the number out of about 30% of evidence tags at the best rounds, and an independent judge preferred the starting reports in 88% of the pairs it decided.

DRe-checking the reported numbers protocol step 3

Measure	Claimed	Audited
MBPP+: how often the grader agrees with the hidden tests	+0.21 (0.417 → 0.625, best round)	At the end of the run the gain is +0.133: across 120 checks (40 tasks, 3 seeds), the grader goes from wrong to right on 36 and from right to wrong on 20 (p = 0.044). That puts agreement at about 0.55, the same as the plain LLM judge alone
MBPP+: a grader that always says 'fail'	-	about 0.70, higher than the evolved grader (our estimate, from a pass rate near 0.3)
MBPP+: gain in task score	106% of the ground-truth gain (best round)	End of run: ground truth +0.317, evolved grader +0.385, collapsed grader +0.315
Spider: gain in task score	110% (best round)	End of run: ground truth +0.010, evolved grader +0.048, collapsed grader +0.051
Report writing: gain in task score	88% (best round)	End of run: rubric +0.240, evolved grader +0.210, collapsed grader +0.252

EHow this audit was done

What we did · paper-level · 2026-10-31

Read the paper and the released code (commit 0f14e91). No run data or logs are released, and the report-writing data is missing, so we could not rerun anything.
Recomputed each gain from the paper's own tables at the end of the run (the average of the last ten rounds), for runs trained with ground truth, with the evolved grader, and with a grader that has collapsed.

Deviations from the original setup

We did not rerun anything; every number comes from the paper's tables or our arithmetic on them.
The paper measures the grader's collapse directly only for report writing; for MBPP+ it says the grader stopped checking anything useful, without giving a number.
With three seeds per run, a difference of 0.07 between runs cannot be ruled out.

Author response: emailed 2026-11-03 — no response yet

CORAL

holdsEvidence in paper L0

CORAL: Towards Autonomous Multi-Agent Evolution for Open-Ended Discovery · arXiv 2604.01658 · 2026

The two released records are real: our independent check reproduces both. This is a case where a fixed evaluator can confirm progress, because the score is exactly the goal. But the standard graders CORAL reuses accept fake scores, and hiding the grader files from the agents does not prevent this. The claims on systems tasks cannot be checked, because those solutions are not released.

AWhat the paper claims

Long-running agents that communicate through shared memory beat the previous best results on 8-10 of 11 optimization tasks, and cut the cycle count on Anthropic's kernel benchmark from 1363 to 1103.

BEvaluators in the loop protocol step 1

Evaluator	Role	Type	Shares with policy	Pressure	What it is
`task_grader_feedback`	selection	fixed	none	direct	Programmatic graders for each task: math constructions checked numerically, cycle counts on a fixed simulator, fixed workloads for systems tasks. Agents run the grader after every submission and see the score and feedback; the grader files are hidden from them.
`task_grader_best_within_budget`	reporting	fixed	task_grader_feedback	—	The final score is the best score the same grader gave within the evaluation budget, averaged over 4 trials. There are no held-out workloads.

Risks to check in this loop

The loop optimizes directly for a fixed score, so it can learn to pass without doing what the score is meant to check.
The agent can see the evaluator's score and feedback, so it can tune against the evaluator.
Tasks used to pick versions also count toward the reported result.

CEvaluator diagnostics protocol step 2

Authors released: final weights. Our reproduction provides final weights.

Diagnostic	Status	Finding
2 telling good from better	required, needs artifacts the authors did not release
4 rewarding the wrong thing	run: evaluator fails	The graders accept fake scores. The Erdős grader trusts the bound the submission reports, so a claim 1e-4 better than the actual construction passes. The kernel grader runs the submission inside the simulator's process, so ten lines of code can set the cycle count to 900. Both weaknesses come from the original graders; CORAL's released solutions use neither.
5 own score vs. truth	run: evaluator holds	For the two released solutions, the reported score matches our independent recomputation: here the score is exactly the goal.
7 what the tests don't cover	run: inconclusive	The systems tasks use fixed workloads with no held-out set, and their solutions are not released, so we cannot test whether they generalize.

Other measurements

grader provenance	The Erdős grader comes from OpenEvolve's grader for AlphaEvolve problems (also copied into SkyDiscover and other projects). The kernel grader keeps the setup of Anthropic's original take-home, where the submitted code runs in the same process as the simulator.
margins	Many results beat the previous best only in the 4th or 5th decimal place (circle packing 2.6360 vs 2.6359; the Erdős row prints 0.38089 vs the previous 0.38088).
systems tasks	Five systems tasks are scored on fixed workloads, with no held-out workloads and no released solutions.

DRe-checking the reported numbers protocol step 3

Measure	Claimed	Audited
Erdős minimum overlap (lower is better)	0.38089 (the table's previous best 0.38088; AlphaEvolve 0.380924)	The released solver gives 0.3808703, and our independent computation gives the same value. The construction meets the problem's constraint to within 6e-10
Kernel builder (cycles)	1103	1103 on all 30 new inputs, all correct; it does not touch the simulator
Fake Erdős submission: the released construction, with a reported bound 1e-4 lower than it actually achieves	-	Accepted and scored 0.38077: the grader uses the reported bound without checking it
Fake kernel submission: the released 1103-cycle kernel plus about ten lines that set the simulator's cycle counter to 900	-	Accepted as 900 cycles, with correct output and 'NEW RECORD'

EHow this audit was done

What we did · empirical · 2026-10-18

Re-ran the released Erdős solver and checked its construction without the grader, by computing the bound directly with an exact integral.
Ran the released 1103-cycle kernel on 30 new random inputs, and checked that it does not touch the simulator.
Submitted two fake entries to the task graders to see whether they can be fooled, and traced both graders back to the projects they were copied from.

Deviations from the original setup

We checked only the two released solutions; the other 9-11 tasks have no released solutions.

Author response: issues filed 2026-10-30 Human-Agent-Society/CORAL/issues/287 algorithmicsuperintelligence/openevolve/issues/562

MASS (Multi-Agent Self-Supervision)

not supported (paper's own data)Evidence in paper L1

Recursive Self-Improvement through Multi-Agent Self-Supervision · arXiv 2610.12176 · 2026

The model judges its own progress, and frontier LLM judges confirm it on 3 synthetic tasks. On public benchmarks with right answers, the model does not improve. One run even rewrote its simulator to make the task trivial, and the outside judges still preferred it.

AWhat the paper claims

One model acts as the worker, the workflow designer and the evaluator, and improves itself round after round: its win rate against the starting model rises from 53.9% to 69.9%, and its evaluator accuracy from 0.73 to 0.93.

BEvaluators in the loop protocol step 1

Evaluator	Role	Type	Shares with policy	Pressure	What it is
`self_pairwise_comparisons`	selection	co-evolving	weights	direct	The model compares pairs of its own work; the comparisons guide the search over workflows and rank the examples used for fine-tuning.
`validation_loss_checkpoint`	stopping	fixed	none	none	Lowest validation loss picks the checkpoint.
`frontier_llm_judges`	reporting	fixed	—	—	GPT-5.5 and Claude Opus 4.8 compare pairs of outputs on 3 synthetic test tasks; plus public agent benchmarks.

Risks to check in this loop

The grader is the model being improved (or an earlier version of it), so it may miss the same mistakes the model makes.

CEvaluator diagnostics protocol step 2

Authors released: paper only.

Diagnostic	Status	Finding
1 telling good from bad	required, needs artifacts the authors did not release
3 saturation	required, needs artifacts the authors did not release
4 rewarding the wrong thing	run: evaluator fails	From the paper (Appendix G.3): a run that rewrote the simulator to make the test trivial was preferred by the outside judges 5 times out of 6.
5 own score vs. truth	run: inconclusive	From the paper: the 'evaluator accuracy' of 0.73 → 0.93 is agreement with the outside LLM judges, not with correct answers.
6 drift and self-preference	required, needs artifacts the authors did not release

Other measurements

in loop evaluator	The 'evaluator accuracy' of 0.73 → 0.93 is agreement with the same outside LLM judges, not with correct answers.
paper reported	In Appendix G.3, the agent rewrote a robotics simulator so that the task always succeeded, and the outside judges preferred that run 5 times out of 6.

DRe-checking the reported numbers protocol step 3

Measure	Claimed	Audited
Win rate against the starting model, 3 test tasks (first round)	53.9%	97 wins and 83 losses over 30 pairs judged 6 times each; by task 29-31, 27-33 and 41-19
SWE-bench Verified	-	62.6 → 61.1 (the paper's own numbers)
Terminal-Bench 2.0	-	37.5 → 35.2 (the paper's own numbers)
MLR-Bench	1.58 → 2.40	the paper's own sign test over 12 task averages gives p = 0.23

EHow this audit was done

What we did · paper-level · 2026-10-11

Read the paper and its appendix. No checkpoints, outputs or judge decisions are released, so we could not rerun anything.

Deviations from the original setup

None.

Author response: responded 2026-11-03 — The authors confirm that the evaluator accuracy is agreement with Opus 4.8 and GPT-5.5 on the workflow decisions made during training. The training tasks have no correct answers, so agreement is used instead; it is measured over the nine training tasks and two cycles, and the evaluators have not been tested on the benchmarks. They attribute the drops on SWE-bench Verified and Terminal-Bench to training only on research tasks, and expect that software-engineering training tasks would help. They plan to look into releasing the checkpoints on Hugging Face.

Our note: This confirms that the evidence that the evaluator improved is agreement with the same outside judges that report the win rate, not a check against correct answers. The explanation for the benchmark drops is plausible but untested; even on MLR-Bench, a research benchmark, the paper's own sign test finds no significant gain (p = 0.23).

autoresearch (Karpathy)

holds, overstatedEvidence in paper L0

autoresearch: AI agents running research on single-GPU nanochat training · GitHub

The overall gain is real: it holds with 30 minutes of training and on other text. But most of it comes from just two of the kept edits: halving the batch size and making the model deeper. The 5-minute score is too noisy to tell whether most of the other kept edits helped at all. It also overvalues edits that just fit more training steps into 5 minutes, and it undervalues the best change: the deeper model gained almost nothing at 5 minutes, and on our GPU it would have been rejected, yet it helps most at 30.

AWhat the paper claims

A coding agent improves a small language-model training script on its own: over a night of 5-minute experiments, the kept edits lower the validation loss step by step, as shown in the progress chart at the top of the repository's README.

BEvaluators in the loop protocol step 1

Evaluator	Role	Type	Shares with policy	Pressure	What it is
`val_bpb_5min`	selection	fixed	none	direct	The validation loss (bits per byte) after a 5-minute training run with one seed. An edit to the training script is kept if it lowers this number and undone otherwise.
`val_bpb_5min_progress`	reporting	fixed	val_bpb_5min	—	The same 5-minute validation loss on the same validation data, plotted across the kept edits as the progress chart.

Risks to check in this loop

The loop optimizes directly for a fixed score, so it can learn to pass without doing what the score is meant to check.
The agent can see the evaluator's score, so it can tune against the evaluator.
Tasks used to pick versions also count toward the reported result.

CEvaluator diagnostics protocol step 2

Authors released: training logs.

Diagnostic	Status	Finding
2 telling good from better	run: evaluator fails	Changing only the random seed moves the 5-minute loss by about 0.0037. 21 of the 22 kept edits improved it by less than that, so the 5-minute score can't tell whether they really helped.
4 rewarding the wrong thing	run: evaluator fails	It rewards edits that squeeze more training steps into 5 minutes. Halving the batch size (the agent's own commit says it was 'for more steps in 5 min') lowers the loss by 0.050 at 5 minutes but by only 0.019 at 30. How much it helps also depends on the GPU: 0.012 on the original H100 SXM and 0.050 on our H100 PCIe.
5 own score vs. truth	run: inconclusive	The gain of the whole chain holds up: the loss drops by 0.054 at 5 minutes and by 0.043 at 30. But most individual edits do not: the last seven edits we measured together lower the loss by 0.010 at 5 minutes and by only 0.001 at 30. And the deeper model points the wrong way at 5 minutes: slightly worse on our GPU, yet the largest gain at 30.
7 what the tests don't cover	run: evaluator holds	The chain's 30-minute gain also holds on other kinds of text (TinyStories -0.055, WikiText -0.059).

Other measurements

fitting more steps	Edits that fit more training steps into 5 minutes keep only about 40% of their 5-minute gain after 30 minutes of training, and all of that comes from halving the batch size. Small tweaks to the training schedule are within noise.
hardware	The same edit is worth different amounts on different GPUs: halving the batch size lowers the loss by 0.012 on the original H100 SXM but by 0.050 on our H100 PCIe, and the deeper model is kept on the SXM but would be rejected on the PCIe.

DRe-checking the reported numbers protocol step 3

Measure	Claimed	Audited
Validation loss at 5 minutes, start to final	a chain of 22 kept edits	1.0648 → 1.0106 (-0.054). With only the seed changed, its standard deviation is 0.0037; 21 of the 22 kept edits improved it by less than that
Validation loss at 30 minutes, start to final	-	0.9578 to 0.9148 (-0.043)
Change on other kinds of text at 30 minutes (TinyStories / WikiText)	-	-0.055 / -0.059
Halving the batch size	-0.050 at 5 minutes (our rerun on the PCIe GPU)	-0.019 at 30 minutes
A deeper model (depth 9)	kept on the original H100 SXM (-0.004 at 5 minutes)	+0.005 at 5 minutes on our PCIe GPU, so it would be rejected; -0.021 at 30 minutes, the largest single gain
The other 7 edits we measured, together	-0.010 at 5 minutes	-0.001 at 30 minutes

EHow this audit was done

What we did · empirical · 2026-10-13

Re-ran 37 configurations from the run Karpathy logged, on an H100 PCIe GPU (his run used an H100 SXM): the starting script, the final script and versions along the chain of kept edits.
Measured how much the 5-minute score changes when only the random seed changes, and compared each logged kept edit with that amount.
Trained the versions along the chain for 30 minutes instead of 5, and also tested those models on other kinds of text (TinyStories, WikiText).

Deviations from the original setup

H100 PCIe instead of the original H100 SXM (fewer training steps fit in 5 minutes)
The 30-minute runs use one seed each, and we did not measure their seed-to-seed noise, so small differences at 30 minutes may not be real
We re-ran 11 of the 23 kept versions one by one; the others were not measured separately

Author response: emailed 2026-11-03 — no response yet

The AI Scientist v2

evaluator failsEvidence in paper L0

The AI Scientist-v2: Workshop-Level Automated Scientific Discovery via Agentic Tree Search · arXiv 2504.08066 · 2025

Humans did the real checking: the authors chose which ideas and runs to submit, and workshop reviewers judged the result. The system's own auto-reviewer can't tell the papers apart; it gave all three the same score. With only three papers, this is a case study.

AWhat the paper claims

An AI-written paper passed peer review at an ICLR 2025 workshop.

BEvaluators in the loop protocol step 1

Evaluator	Role	Type	Shares with policy	Pressure	What it is
`llm_reviewer`	selection	fixed	—	filter	An automated reviewer (GPT-4o) scores the generated papers; the scores are used to pick what is kept.
`human_workshop_review`	reporting	fixed	—	—	Human reviewers at an ICLR 2025 workshop, on the three submitted papers.

Risks to check in this loop

The start and end were not measured the same way.

CEvaluator diagnostics protocol step 2

Authors released: final weights.

Diagnostic	Status	Finding
1 telling good from bad	run: evaluator fails	The auto-reviewer gave all three papers a 7 and accepted them, including the one human reviewers scored 3, 3 and 3. Using the conference criteria, it ranked the paper humans liked least the highest. Only three papers.
2 telling good from better	required, needs artifacts the authors did not release
4 rewarding the wrong thing	to do: required and possible, not run yet
5 own score vs. truth	required, needs artifacts the authors did not release

Other measurements

in loop evaluator	It accepted even the paper that all three human reviewers scored 3. Using the conference criteria, it ranked the paper humans liked least as the best.

DRe-checking the reported numbers protocol step 3

Measure	Claimed	Audited
Average human reviewer score (3 papers)	one accepted	6.33 / 4.67 / 3.0
Auto-reviewer, workshop track	-	7 / 7 / 7, all Accept
Auto-reviewer, conference track	-	3 / 3 / 4: ranks the paper humans liked least the highest

EHow this audit was done

What we did · empirical · 2026-10-05

Re-ran the system's auto-reviewer on the three released workshop submissions and compared with the human scores.

Deviations from the original setup

None.

Author response: not contacted — this entry does not question the result the paper reports (an AI-written paper was accepted at the workshop)

Appendix: do our diagnostics measure what they claim? A check on loops with known truth

We ran our own simulated self-improvement loops with Qwen2.5-7B (iterated DPO, 4 rounds, 2 seeds) and kept every round's answers, the scores the loop's evaluator gave them, and the scores of a separate reward model that the loop never sees. That hidden reward model is the truth here, measured in standard deviations of the starting model's scores. D1 and D2 are how often the evaluator picks the truly better of two answers to the same prompt, counting only pairs whose true scores differ by at least 0.25 standard deviations; D2 counts only pairs where both answers score above the starting model's average.

Diagnostic (rounds 1–4, mean of 2 seeds)	fixed capped reward model	co-evolving model judges itself	hybrid half each
D1 picks the better of two answers to the same prompt (0.5 is chance)	0.75–0.87	0.60–0.66	0.73–0.81
D2 the same, but only when both answers are above average	0.50	0.57–0.60	0.56–0.60
D3 portion of answers that all get the same score	0.52–0.72	0.00–0.01	0.00
D5 how often the loop's own evaluator rates a round's answers above round 1's	—	59–65%	54–56%
D6 judge accuracy on a fixed answer pool, round 0 → 4	—	0.67 → 0.63	0.68 → 0.65

fixed: the hidden reward model, with scores capped at the starting model's average, so it cannot tell apart answers above that level. co-evolving: the model judges its own answers. hybrid: half the model's own judge, half the capped reward model.

What the diagnostics show

D2 catches the fixed evaluator's ceiling exactly. Its D1 looks good, but among answers above the ceiling it does no better than chance (0.50). And in the rounds where true quality is higher, more answers get the same top score (D3 0.71-0.72, against 0.52-0.55 in the other rounds).
The self-judge is weak throughout (D1 about 0.63), and it keeps saying the answers are better than round 1's whether true quality goes up or down (D5).
Among the best answers, the hybrid is no better than the self-judge alone (D2 0.56–0.60 against 0.57–0.60): above the cap, the capped half adds nothing.
D6 was measured separately, by having every round's judge score the same fixed set of answers, so changes come from the judge alone. The self-judge gets worse round by round, 0.668 → 0.632 (-0.037 [-0.062, -0.012]), and it does not favor its own round's answers.

Caveats

True quality per round varies a lot between the two seeds (the 95% CI over prompts is about ±0.1, but in the same round the two seeds can differ by up to 2 standard deviations), so we only state what holds for both seeds.