Clinical Epistemology · folder 11 × Meta-Harness

The layer nobody optimized

A Stanford, MIT and KRAFTON group automated the design of the harness — the code that decides what an LLM stores, retrieves, and is shown. Their method is directly runnable on this portfolio's existing machinery. But their objective function is a scalar accuracy score, and a scalar accuracy score prices abstention at zero. That is the seam where this portfolio has something the paper does not.
Source: Lee, Nair, Zhang, Lee, Khattab & Finn, Meta-Harness: End-to-End Optimization of Model Harnesses, arXiv:2603.28052v1, 30 March 2026 (preprint, not peer reviewed) Status: v0.1 convergence note Created: 2026-08-19 Companion to: 04_Folder11_ECIF_Convergence_Map.html · 02_Decision_Theory_and_Strategy_Update.md · 06_Agent_Memory_Survey_Review.html
01

Three bands, and which one each side owns

Meta-Harness freezes the model and searches the harness. It leaves the objective fixed, because on a benchmark the objective is given. In clinical work the objective is the hard part, and it is the band this portfolio has spent two years specifying.
Meta-Harness
searches this
→

An agentic proposer (Claude Code, Opus 4.6) rewrites the harness code, scores it, and reads the raw traces of every prior attempt off a filesystem.

Band 1 · frozen
The model
Held constant throughout. GPT-OSS-120B, GPT-OSS-20B, Claude Opus 4.6, Claude Haiku 4.5. In this portfolio: whichever OpenRouter engine the harness happens to call.
both sides treat this as a commodity
Band 2 · the searched layer
The harness — what is stored, retrieved, and shown
Meta-Harness's finding: on TerminalBench-2, six harnesses around one frozen Claude Haiku 4.5 span 27.5% to 37.6%. Ten points, same weights. The ablation says the whole trick is giving the proposer raw execution traces (50.0 median) rather than model-written summaries of them (34.9, no better than bare scores at 34.6).
Meta-Harness: automated search Portfolio: BRIDGE master prompt, CRRF nine steps, NBA rules — all hand-written
Band 3 · fixed in the paper, specified here
The objective — what counts as a better harness
Meta-Harness uses accuracy and pass rate. Both score I cannot determine this identically to a confident wrong answer. The portfolio's T0–T5 earned-authority ladder is a utility function over exactly this, and folder 11's decision-theory note already names the formal machinery: maximum expected utility with an explicit abstain action, plus value of information.
Portfolio contribution Neither side has implemented it
The portfolio
owns this
←

CRRF's nine steps, the six-tier ladder, and BRIDGE's certainty bands are a specification of what a good answer is — which is the input a search needs and does not have.

The identity that joins them. Meta-Harness has a method for improving harnesses and an impoverished notion of what "better" means. This portfolio has a rich notion of what "better" means and no method for improving harnesses. They are complementary halves of the same problem, and each one's contribution is the other's stated limitation.
02

The claim worth writing up

Thesis
Harness search optimizes whatever you score. If the objective does not price abstention, automated harness engineering will engineer the abstention out — and the score will improve while it happens.

This is not a hypothetical failure mode, it is the expected behaviour of the method. Meta-Harness's proposer reads the traces of every failed candidate and rewrites the code to raise the score. On any benchmark where a refusal is scored as a miss, a harness that refuses less will score higher. The search will find that. It will look like progress on the metric, and it will be a regression in exactly the property this portfolio exists to protect.

The corollary is the useful part. The T0–T5 ladder is a candidate objective function, not just a presentation device. A harness that earns T5 and is right should beat one that claims T5 and is wrong by a wide margin; a harness that correctly drops to T0 should beat both. Score a search that way and the discovered harnesses should differ — measurably, and in a direction that is publishable either way.

And 09_aberrant-labs/ is already the adversarial set that would demonstrate it: 20 reports carrying 40 implanted physiologic impossibilities with ground truth on every one, where the correct behaviour is refusal rather than interpretation. An accuracy-maximising search, given those cases, would learn to answer them confidently. A tier-weighted search should not.

03

Construct-by-construct crosswalk

Same direction convention as the ECIF map: which side, if either, repairs the other.
same — the two describe one thing MH repairs portfolio — the paper supplies something missing here portfolio repairs MH — this repository supplies something the paper lacks gap in both — neither has it
Meta-Harness constructPortfolio instantiationDirectionNote
Harness — code deciding what is stored, retrieved, presented BRIDGE_Master_Prompt.md + its JSON schema; the CRRF nine steps; NBA_RULES_v1.0.md same The portfolio has been building harnesses for two years without the word. Naming it makes the layer countable.
Freeze the model, vary the harness The multi-model OpenRouter panel in 05, 08, 09 and the BRIDGE test harness — one fixed prompt, many engines same Already the QA method here, run in the opposite direction: vary the model, hold the harness. Both axes are informative.
Filesystem of raw prior traces, queried with grep/cat test-harness/*_raw.jsonl; 05/…/reviews/*_red_team.txt; _QA_detection_review.md same Already produced and never fed back. The paper's key ingredient is an artifact this repository writes on every run and then archives.
Scalar objective (accuracy, pass rate) CRRF T0–T5 earned-authority ladder; BRIDGE certainty bands (90–100 / 70–89 / 40–69 / 15–39 / 0–14) portfolio repairs MH The load-bearing row. The paper has no way to price a correct abstention; the ladder is precisely that pricing, and it is already written down.
Ablation over information access (their Table 3: scores / summaries / full traces) No gate ablation exists — the nine steps are asserted, not measured MH repairs portfolio The portfolio's own red team found "no outcome evidence yet exists for the gates." This is the experimental design that would generate it.
Overfitting is inspectable in code space (brittle if-chains are visible) Gates are readable prose; every recommendation ships support, counter-evidence, provenance and reversal conditions (step 8) same Both argue legibility beats weight-space opacity. The paper reaches it as a convenience; the portfolio treats it as the requirement.
Transfer to four held-out models (+4.7 points average) Cross-engine agreement scoring in bridge_score_harness.py same Same measurement, different name. Note the paper's own softness here: gains run +8.7 to +1.6, and the "five held-out models" average includes the search model.
Environment bootstrapping — snapshot the state and inject it before the loop starts CLAUDE.md R5: "read CHANGELOG.md first, then OPEN-ITEMS.md" MH repairs portfolio Same instinct, executed by hand against a 100 KB append-only log. Addressed 2026-08-19 by STATE.md and _reference/tools/build_state.py.
Proposer that rewrites the artifact and re-scores it Red-team / white-team multi-model review (Grok, GPT, GLM, Kimi) across 05, 08, 09 and the BRIDGE cases MH repairs portfolio The panel critiques and a human applies the fix. It never closes the loop by rewriting and re-scoring itself. That closure is the whole method.
No utility or values component anywhere Also absent — flagged in the ECIF map as a gap in both, and named but unimplemented in 02_Decision_Theory gap in both Confidence is only the probability half of a recommendation. Value of information is the formal answer and remains prose in this repository.
No lifecycle model for the discovered harness Also absent — the same Version/Owner/MonitoringMetric gap flagged for the knowledge graph gap in both A harness discovered against March guidelines silently rots when the guideline changes. Neither side models that.
Search and evaluation on the same 89 tasks, no held-out split holdout_cases.json — a real held-out split, but n = 5 gap in both The paper is underpowered against overfitting and discloses it. This repository has the right design and not enough cases to run it. Different failures, same consequence.
04

What is already here, and what is genuinely missing

Meta-Harness needs three things: a frozen model, a scoreable evaluation, and a harness that can be varied. Two and a half of those are built.

Built, today

No new engineering required
The harness06_extraction-heuristic/BRIDGE_Master_Prompt.md — six stages, a DEVELOPER CONFIGURATION block, an appendix of worked examples, and a scoreable JSON schema. This is a harness in the paper's exact sense.
The evaluationtest-harness/bridge_score_harness.py — 301 lines, pure stdlib, field-level scoring against ground truth, scope-guard cases, cross-engine consistency, machine-readable report.
The traceslast_run_raw.jsonl, run_glm_raw.jsonl, run_clin_raw.jsonl — "every raw response, for audit." The paper's decisive input, already on disk.
The abstention set09_aberrant-labs/ — 40 implanted errors with ground truth, plus the two scope-guard holdout cases where the correct output is a refusal.
The ablation switch--strip-appendix already exists in the harness runner, to prove the held-out cases do not lean on the embedded examples. That is one gate ablation, already implemented.

Missing, and honestly so

The real work, in order
Case countFive holdout cases will overfit inside three iterations. A search needs roughly 40 to 60 scored cases with a genuine held-out split. The generators exist (05 packets, 08 extraction keys, 09 error keys, Patients/); the cases do not. This is a data task, not an AI task, and it is the first one.
A tier-weighted scoreThe scorer currently reports field agreement. It does not price a correct abstention above a confident error. Until it does, any search run against it will drift the wrong way.
The outer loopNothing reads the raw JSONL and proposes a revised prompt. This is the smallest piece of the three and should be built last, after the objective is right.
Contamination disciplineThe paper audits by regex for task strings leaking into evolved harnesses. Any loop run here needs the same, and the synthetic cohort makes leakage easy to check and easy to miss.
05

The single move worth making first

Not the search. The ablation — because it answers a question the portfolio's own red team already raised and cannot currently answer.

Gate ablation on BRIDGE

Meta-Harness Table 3, applied to CRRF · turns the nine steps from architecture into finding
1
Expand the holdout set to ~40 cases. Draw from the six scored hospital packets, the aberrant-lab errors, and fresh generations from Patients/. Hold a third back and never look at it until the end.
2
Write the tier-weighted objective. Score each output on the T0–T5 ladder rather than on correctness alone: correct-and-earned > correct-abstention > over-claimed-but-right > confidently wrong. The relative weights are the clinical judgment, and they are Adam's to set — that is the point, not a limitation.
3
Ablate one gate at a time. Run BRIDGE with the sufficiency judgment removed. Then the coherence check. Then both. Then the known-unknowns requirement. Measure what each gate buys under the tier-weighted score and under plain accuracy — the difference between those two curves is the argument.
4
Only then close the loop. With a defensible objective and a set large enough to hold out, point a proposer at the raw traces. If the discovered harness deletes the honest exit, that is a result. If it does not, that is a better one.
Why this is the right first move. The red-team review of CRRF's novelty concluded that the defensible core is the conjunction of the sufficiency and coherence gates, and that no outcome evidence yet exists for any of them. Ablation is how that evidence gets made. It needs no outer loop, no proposer, and no new model — only the scorer this repository already has and more cases than it currently holds.
06

Where this note is weaker than it reads

About the paper

Read before citing any of its numbers
The 6× figureAppears in the paper's opening sentence as a citation to prior work, not as a Meta-Harness measurement. Secondary commentary has already begun attributing it to these authors. Cite the ten-point Haiku spread instead; that one is theirs.
"Five held-out models"The abstract's phrase. Four are held out; the fifth is GPT-OSS-20B, the model the search ran on, and it sits inside the +4.7 average.
Coding resultSearch and evaluation on the same 89 tasks, disclosed. Its reported #2 leaderboard position dated quickly — the board now carries 142 submissions and tops out near 84.7%.
Cost10.0 MTok of feedback context per iteration against 0.026 for the next most generous method, roughly 385×, across ~60 harnesses per run. No dollar figure is reported. This is a capital expense that only pays back on a stable task.

About this note

Its own honest exit
One domainThe traces-beat-summaries ablation is one task type, one proposer, one base model. Generalising it to clinical extraction is an inference this note makes and does not evidence.
Untested claimThat an accuracy-maximising search would delete abstention is an argument from the objective's structure. It is not measured here. Step 3 above is what would measure it, and it could come out the other way.
Appendix unreadThe paper's Appendix B, which specifies the discovered strategies ("Draft Verification", "Label-Primed Query"), was not reachable in the HTML rendering. Those mechanisms are named here and not described.
ScopeThis is a convergence note, not a plan of record. Nothing in §05 has been built, scoped, or costed.