Day 76: the sweep runs as a process, and every row carries its own verdict
Date: 2026-10-05 · Week 14 · Phase 5 Benchmark and optimize
What I added today
Yesterday’s “Tomorrow” was a two-rate sweep smoke: --rates with --arrivals fixed, so the multi-row table, print_table’s split columns and a two-row CSV run in a real process. That’s the shape of the card run, and the burst smoke never reached that half of main.
The sweep smoke. tests/test_toy_script.py gets a second module fixture that runs graphbench.py --split-read --arrivals fixed --rates 200,400 over the same toy checkpoint. The rates are high on purpose: six fixed arrivals 2.5 to 5 ms apart land inside one step, so the split arm still batches. Six tests check that every rate boots split, graphs, eager again in that order, that both rates pad to the same prompt, that every split claim and “same answers” print ok once per rate, that the table prints with its split columns, and that the CSV has one row per rate under the burst CSV’s exact header.
It passed the first time. There was no bug in the sweep. What the run did show was in its output: at both rates the log printed FAIL the arms are comparable and FAIL the tail did not get worse, and the table and the CSV under it printed the ratios with no sign of either failure.
So the row now carries its verdict. In graphbench.py:
claims(delta, args)is the listmainused to build inline: the eight two-arm gates, thensplit_claims.grade(pairs)runs each one and keeps(note, exc or None). It catchesAcceptanceFailureandMeasurementUnsoundand nothing else, so a bug inside a check still crashes the run instead of printing as a FAIL.verdict_columns(graded)givesclaims_ok(a count) andclaims_failed(the failed notes joined with"; "). They go at the end of the row, afterreplay_share.print_tablegets aclaimscolumn that reads held over total:11/13.--sanitynow prints every claim before it exits, not just up to the first FAIL.
tests/test_sweep_verdict.py covers the four functions in-process (12 tests) and three more process tests hold the sweep’s CSV and table to what its log said. Suite 2382 green (34 GPU-gated skips), ruff clean.
Why it matters
The CSV travels without the log. The card run’s output that gets plotted is the CSV. Before today, a 1 rps row whose p99 came from eleven samples, refused by its own gate, read exactly like a clean one. Now claims_failed is non-empty, and a notebook can filter on it.
The table is what a person reads last. The claims print above the table, one block per rate. With four rates that’s about sixty lines before the table, and the table is what goes in the post. 11/13 at the end of a row says “go read the log” where the number is.
Grading once keeps the log and the row honest with each other. The log prints from the graded list and the row is updated from the same list. A second loop over the gates for the CSV could disagree with the first on a gate that reads the clock.
What I learned
- A smoke that passes still has something to say. I wrote the sweep tests expecting a crash like yesterday’s. They went green, and the useful finding was in the output, not the exit code: the table printed numbers its own gates had refused.
- Catch the refusals, not everything.
gradecould have caughtExceptionand been more forgiving. It would also have turned aKeyErrorin a gate intoFAIL the tail did not get worse, which is a wrong claim about the server and not a report of a bug in the harness. There’s a test for exactly that. - New columns go at the end. Day 74’s split columns came after the two-arm ones so an older CSV stays a prefix. Same rule here:
claims_okandclaims_failedcome afterreplay_share, and a test pins them as the last two. - Know what a fixture fails before you count on it. The in-process
_deltacarries emptyCaptureStats, so its graphed arm has no capture and fails the three capture claims. My first count said 12 of 13 and it was 10. The tests name those three rather than hard-coding a number nobody can explain.
Diagram
graphbench-sweep-verdict.png. Top left: before, the log’s FAIL lines and a CSV row that doesn’t mention them. Top right: claims to grade to verdict_columns. Bottom left: what the sweep smoke checks. Bottom right: the table’s claims column and the CSV’s two new columns.
Tomorrow
Every part of the script the card run uses has now run in a process on CPU, except the compile path: both smokes pass --no-compile. Next: a toy smoke with --compile dynamic on CPU (one rate, burst), so CompiledDecode actually builds behind a booted server and the capture records over a compiled forward. If inductor is too slow for the suite on this box, it gets its own marker and runs on demand.
The hardware caveat hasn’t changed: the card run with --split-read --rates 1,2,4,8 is still the first one to book, and now its CSV will say which rows to trust.
Post angle
Day 76 of building an LLM inference engine from scratch. I ran the benchmark sweep as a real process expecting another crash. It passed. The find was in the output: two gates printed FAIL at every rate, and the table and CSV under them showed the ratios as if nothing happened. The CSV is what gets plotted, so now every row carries claims_ok and claims_failed, and the table says 11/13. vLLM and SGLang are the production systems I’m learning from; this one is about a benchmark that can’t hide its own refusals. 2382 green.