Day 75: the benchmark script runs as a process, and its first run finds three bugs
Date: 2026-10-04 · Week 14 · Phase 5 Benchmark and optimize
What I added today
Yesterday’s “Tomorrow” was a toy-weights smoke for graphbench.py itself. Every test of the script so far either checked its arithmetic or called build_one with build_app monkeypatched to a lambda that takes any keyword. Nothing had ever run the script.
nanoserve/toyckpt.py. write_toy_checkpoint(dir, seed=0) writes what ./weights holds, at toy size:
config.jsonwith HuggingFace’s field names (2 layers, GQA 8:2, vocab 256).model.safetensorsunder HuggingFace’s tensor names, with nolm_head.weight, so the loader takes the tied-head alias path the 1B checkpoint takes.tokenizer.jsonandtokenizer_config.json: a byte-level BPE with no merges, so one byte is one id andencodeadds no specials. No EOS token, so a random model can’t end a client at a step nobody chose.
It refuses a directory that already holds any of the four files. The same seed writes the same bytes.
tests/test_toy_script.py. 17 tests. Eleven cover the checkpoint: it loads through ModelConfig.from_json, load_weights and AutoTokenizer.from_pretrained with nothing injected. Six run graphbench.py --split-read --allow-cpu --arrivals burst --max-model-len 1024 once as a subprocess, then check the exit code, the boot order, the padded length, every split claim, the cross-arm answers and the CSV columns.
The three bugs it found, all there since Day 57:
build_onepassedcompile_decodetobuild_app, andbuild_enginehad no parameter for it: aTypeErrorbefore the first arm loaded.build_enginenow threads it through toEngine.build.- The “graphs” arm got buckets and persistent inputs but never
capture_decode, so it would have booted with the graphs off.build_onenow passescapture_decode=graphs. --compiledefaulted to"default", which istorch.compile’s word.CompiledDecodetakesoff | dynamic | staticand raises on anything else. The default is now"dynamic", argparse offers onlyMODES, andoffmaps to no compile.
Unit tests guard all three: the script’s keywords are held to the real build_app/build_engine signatures, the graphed arm must ask for the capture, and the parser rejects --compile default. Suite 2360 green (34 GPU-gated skips), ruff clean.
Why it matters
The card run would have died in its first second. Booking a GPU, typing yesterday’s one command, and getting a TypeError before the first arm loads is the cheap version. Bug 2 was worse: if bug 1 had been fixed alone, the run would have finished, the “capture was used” claim would have failed, and every ratio in the table would have compared two eager servers.
A patched launcher accepts anything. lambda weights, **kw: calls.append(kw) is fine for asserting what the script passes and useless for checking whether the callee takes it. The Day 74 test split == graphs passed with a keyword nothing downstream knew, because both arms passed the same wrong keyword.
A smoke on CPU costs 14 seconds. The script boots three servers over a 2-layer model, pads the prompt with a real HF tokenizer, serves six clients at a width that splits, and writes the CSV. That’s cheap enough to run on every suite pass.
What I learned
- Test a script by running it. An in-process test with a patched
build_appchecks what the script asks for. Only a process checks that what it asks for exists. The three bugs were all at that seam. - A byte tokenizer makes the padding predictable.
split_promptjoins copies with a space, so n copies of the 24-byte prompt are 25n - 1 tokens, and 21 copies (524) is the first count past the 514 floor. The test asserts 524 exactly, not “at least 514”. - Write the checkpoint the way it ships. Leaving
lm_head.weightout of the file sends the toy run through the same tied-head alias path as Llama-3.2-1B. A toy checkpoint with an untied head would test a path the real run never takes. - A FAIL can be the right output. On the six-request smoke, “the arms are comparable” fails with 11 frame gaps against the 20 it wants. That gate is meant to refuse a p99 from a handful of samples, so the smoke asserts the split claims and the answers rather than a clean sheet.
Diagram
graphbench-toy-smoke.png. Top left: the four files and the real calls that read them. Top right: the three bugs and their fixes. Bottom left: the command, the boot order and the padding arithmetic. Bottom right: what the script printed once fixed (CPU, fixture-sized).
Tomorrow
The script now runs end to end, but on CPU it runs --no-compile, so the compile path it passes to the engine has only been checked at the signature level. Next: extend the smoke to a two-rate sweep (--rates with --arrivals fixed) so the multi-row table, print_table’s split columns and a two-row CSV are exercised on a real process. That’s the last part of the script the card run uses that hasn’t been run.
The hardware caveat hasn’t changed: the card run with --split-read --rates 1,2,4,8 is still the first one to book. Now it has actually been run once.
Post angle
Day 75 of building an LLM inference engine from scratch. I ran my benchmark script as a real process for the first time, over a tiny checkpoint written to a temp directory, and it died at boot. Three bugs from Day 57, all hidden by tests that mocked the launcher: an unknown kwarg, a “CUDA graphs” arm that never asked for the capture, and a bad compile default. 14 seconds on CPU found them, before any card time was booked. vLLM and SGLang are the production systems I’m learning from; this was a lesson about the harness, not the kernels. 2360 green.