93% on RepoMod-Bench: porting 200K-line codebases with a proof in the loop

Bosun’s modernization workflow rests on one bet: an agent that rewrites a codebase should be held to what the original program does, and a proof of that behaviour is worth its cost. A proof here is a Parity proof. We record what the original does on a set of cases, down to the files it writes, the processes it starts, services it interacts with, effects, and output, and replay the same cases against the port. In our approach, agents are orchestrated into a structured workflow where they assess, write spec-like proofs, split up the work, review, and integrate.

We wanted to test that bet in public, on code we did not pick, graded by someone else. RepoMod-Bench made that possible: 21 open-source repositories, from 14 lines to 211K, each ported to another language and graded by a hidden test suite. We ran all 21 three times, unattended, and published the transcripts, the proofs and the ported code of all 63 runs. For each case we report the best of the three runs, for the workflow and for our control alike. Three results:

The workflow passes 93% of the hidden tests, and 88% on the twelve repositories of 50K lines and more. The paper’s best agent, Claude Code on Opus 4.5, reaches 48% and 20%. Part of that gap is a newer model, so we reran the paper’s Codex setup on our model: it reaches 74% and 55%. The workflow adds 19 points overall and 33 on the large repositories, where taskwarrior, verible, qalculate and ledger go from 0–69% to 97–100%.

Half of what the proof caught, the hidden tests do not check. Of the 411 behaviour differences the proof surfaced and the agent fixed, 205 are behaviour no hidden test exercises. Ports that pass every hidden test still have 79 open differences on record, among them a wabt port with no wasm2c and no wasm-decompile. A team that has to merge a rewrite gets a replayable list of where the port differs from the original.

A 211K-line C++ calculator, ported to Go, passes every hidden test. A run took about three hours and €19 of inference. Across all 21 cases a run costs €9.50 and takes two hours on average.

The setup

RepoMod-Bench asks an agent to port each repository to another language (C++ to Rust, Go to Python, JavaScript to Rust, and so on) and grades the result with a hidden test suite derived from the original’s own tests. The paper ran four agents: Claude Code and OpenCode on Opus 4.5, Codex CLI and OpenCode on GPT-5.2, one run each with four hours per case.

We ran the Parity workflow on all 21 cases, three times each, with the same four-hour limit and one model throughout, GPT-6.1 Sol.

Sol is newer than the paper’s models, so we also ran a control: the paper’s Codex CLI setup with the same command and prompt, three runs per case, and the model swapped from GPT-5.2 to Sol. As in the paper, Codex ran in its default configuration. It had a three-hour limit and a newer Codex than the paper used. The control shows what the model alone adds to the paper’s result.

The results

	All 21 cases	12 cases of 50K lines and more	Cost per run	Time per run
Best agent in the paper (Claude Code, Opus 4.5)	48%	20%	–	39 min
The paper’s Codex CLI, GPT-5.2	30%	4%	–	20 min
The same Codex CLI setup, Sol	74%	55%	€0.80	9 min
Parity workflow, Sol	93%	88%	€9.50	2 h

Our two rows take the best of three runs per case; the paper’s rows are its single runs. Most of the jump from the paper is the model: Sol takes the paper’s Codex setup from 30% to 74%. The workflow adds 19 points on top, and 33 points on the twelve large repositories. Picking the best of the paper’s four agents for each case gives 58% overall and 27% on the large ones. Below 25K lines there is nothing to add, because both setups pass every test on all nine cases.

The twelve large cases, by size:

Repository	Port	Lines	Parity workflow	Codex on Sol	Best in the paper
ledger	C++ → Go	50K	97%	62%	9%
wabt	C++ → Rust	55K	100%	89%	52%
taskwarrior	C++ → Rust	55K	100%	69%	11%
lightningcss	Rust → Go	62K	93%	77%	29%
bc	C → Rust	117K	100%	100%	26%
hugo	Go → Rust	122K	76%	70%	46%
jq-gojq	C → Go	147K	100%	100%	51%
pdfcpu	Go → Rust	160K	82%	6%	48%
uncrustify	C++ → Rust	162K	48%	0%	0%
prettier	JavaScript → Rust	175K	55%	21%	9%
verible	C++ → Rust	191K	100%	0%	20%
qalculate	C++ → Go	211K	100%	64%	20%

The workflow is ahead on ten cases and level on the other eleven. Six of the twelve large ports pass every hidden test. The result does not depend on picking a lucky run: on nine of the ten (ledger, wabt, taskwarrior, hugo, pdfcpu, uncrustify, prettier, verible, qalculate) the workflow’s worst run beats the control’s best. For ledger that is 97% against 62%, for taskwarrior 99% against 69%, for qalculate 97% against 64%. lightningcss is the one case with a wide spread: 59% to 93% depending on how the plan cut it.

Against the paper, we match or beat the best published agent on all 21 cases. Two of them, pdfcpu and uncrustify, are regraded: their hidden tests open sample files from inside the ported tree, and the task does not ask for those files, so none of our agents copied them. Graded as-is, the ports score 6% and 0%. With the sample files in place, the same frozen ports pass 82% of pdfcpu’s tests (the paper’s best is 48%) and 48% of uncrustify’s (0% for the paper’s agents). The tables and the chart use the regraded scores for the workflow.

How the workflow works

Bosun modernises mission-critical software: it inventories what a system does, migrates it one slice at a time, and checks each slice against the original before the next one starts. Parity is the tool that does the checking. You describe cases (a command, its inputs, its environment), Parity runs them against the original under audit and records what it can observe: stdout, stderr, exit code, the files the program reads and writes, the processes it starts, the signals it receives, the state it leaves behind. Running the same cases against the port gives a typed list of differences, each with the original’s recording next to it. A proof is such a set of cases plus the recorded expectation; it replays, so a reviewer can run it again later.

The Parity workflow is the one Bosun’s modernization pipeline is built on, pointed at a whole repository instead of a module. Four moves, repeated:

1. Survey and plan. Read the original’s own tests and documentation, list its feature areas, cut the port into slices, and give each area one owner.
2. Prove, then implement. For each slice, write the Parity proof first. The cases take their inputs from the original’s own tests; the expectation is what Parity records the original doing, which covers more than those tests assert. Then an agent implements the slice in a fresh context and runs the proof against the port.
3. Repair with evidence. A mismatch arrives as a typed difference: this file, this byte range, this exit code, with the recording of what the original did. Repair gets two passes, then the loop moves on to the next slice whatever the result.
4. Conform and finalize. One pass over the whole port against every proof, then a final build, a full proof run, and a check that the shared audit policy is unchanged.

We tuned the workflow on three cases (lightningcss, wabt, hugo), froze the manifest by hash, and then ran the other eighteen. On those eighteen the workflow passes 93% against the control’s 73%. The manifest sits at the root of the dataset.

What a proof finds that a test suite does not

A test suite checks what its authors asserted. A proof compares the port with the original on what Parity can observe. To measure the difference we classified the repair step’s fixes by hand against a written rubric, which gave 411 behaviour differences the proof surfaced and the agent fixed. For each one, a reviewing model read the case’s hidden tests and graded whether any of them would have noticed.

Half of them (205) are behaviour the tests never exercise. Another 37% are reached only by an exit-code check: the test knows the command failed, not what it said. 13% are covered. The differences the tests miss fall into familiar classes:

• Output format. tokei’s YAML output printed emoji filenames as \U escapes. httpie-xh showed a download longer than an hour as minutes instead of 1:02:03.
• Diagnostics. tokei --outptu stopped suggesting --output. prettier’s error for <br></br> lost its code frame.
• Parsing edge cases. gitleaks opened .tgz files as archives when the original does not. toml hit Python’s recursion limit on a deeply nested document.
• File side effects. tokei applied a parent repository’s .gitignore across a nested .git boundary. gitleaks reported its own baseline file as a leak.
• Command-line arguments. tokei accepted -o json -o yaml where the original rejects a repeated flag.
• Versions and state. httpie-xh compared versions as strings, so 3.10.0 counted as older than 3.9.0 when deciding on update warnings and session migration.

The differences that stayed open are on record too. Across the ports that pass 100% of the hidden tests, the proofs list 79 behaviour differences that are still open, and no hidden test asserts on any of them. qalculate’s port does not load the definitions and readline config the original reads at start-up, so a user’s customisations are ignored. A JMESPath expression nested 130 levels deep errors in the port and succeeds in the original. Two runs were missing whole tools: one wabt port had no wasm2c and no wasm-decompile, one verible port had no project tool, Kythe extractor or language server. The hidden suites run two of wabt’s tools and five of verible’s, so both ports scored 100%.

Each of these differences is a replayable comparison against the original, with the recording attached. A reviewer deciding whether to merge a port can read that list and rerun it.

Language runtimes add noise to an audit. Go sends itself scheduler signals, Rust probes the CPU at start-up, both read /proc. A fixed policy excludes the known shapes, and an agent can add a narrow exclusion of its own, scoped to one proof and one resource with a stated reason. Every exclusion is in the dataset. Five of the 63 runs tried to edit the shared policy file instead. The final integrity check caught all five, and they are marked as such in the data.

Cost and time

The 63 runs cost about €600 of inference: €9.50 and two hours of wall time per run on average, so three runs of a case come to about €29. The twelve large repositories average about €13 and just under three hours per run. The control costs €0.80 and nine minutes per run. No run came near its spending cap. Eight cases ran for more than three of their four hours, and one run hit the time limit.

Modernizing at scale

The workflow is a simplified version of Bosun’s approach. There are several important features in Bosun that help improve quality, operation, and fill the remaining gap:

• multiple workflows from assessment to pull request Steps in the benchmark are actually full (customizable) workflows in Bosun, and slices produces one or more pull requests, incrementally and validated. Work moves over a kanban board like a factory.
• code intelligence In the workflow, agents relied on shell tools only for code discovery. Builds a deep semantic graph both for agents and additional validation and understanding.
• replay Records real traffic (and/or other behavior) and replay it like a VCR against slices. This helps agents fill the gap, and engineers release with confidence.
• project memory Workflows can learn within itself and from previous runs.
• strategy and code-quality Are planned stages in the platform, but only lightly touched in the benchmark. No modernization is the same, the benchmark left that detail out.
• model choice Use any (cheaper, faster, open/closed, or better) that fits the organization, problem, and profile.

If you have a modernization to do

Take qalculate as the reference point: 211K lines of C++, ported to Go, every hidden test passing in the best of three runs, at about three hours and €19 of inference per run. At that price a rewrite you have deferred for years is cheap to attempt. The port also arrives with its proofs: a replayable record of where it matches the original and where it still differs, down to the file and the byte. Where the workflow could not close a difference, the record says so.

The data

The whole run is public as bosun-ai/repomod on Hugging Face. It holds fifteen tables you can browse in the viewer (per-case results, each hidden test, per-step cost, the Parity records, the classified fixes and exclusions, the coverage grading, the full agent transcripts), the workflow manifest, and one archive per run with the proofs, verify reports and the ported code as submitted. About 60 GB of evidence, 1.6 GB compressed, with a manifest of hashes. If you doubt a number in this post, you can recompute it.

If your team has a port it keeps putting off, book a demo.