The Oxbox Survey

The Oxbox Survey catalogs the free and cheap models on the public gateways, tries a few of them, and reports what it took to get a usable code review out of each one. Every model here was given the same job: review public code, through oxbox. More about the survey.

This edition asks one question: what does a reader give up, and what do they pay, by running a cheap model with a check instead of a frontier model on its own? Four frontier models ran the same three fixtures the cheap and free models have run, and the bills are side by side.

This week's highlights

• Four frontier models invented findings on a 6 KB file with two known defects, the same way the cheap ones did. anthropic/claude-opus-5 emitted 9 findings and 7 were wrong, openai/gpt-5.6-terra-pro emitted 5 and 4 were wrong, google/gemini-3.1-pro-preview emitted 2 and both were wrong.
• The cheap model with a check was cheaper than the frontier model without one. z-ai/glm-5.3-flash plus an Opus 5 check cost 16 cents for one verified finding on the 6 KB file; Opus 5 alone cost 24 cents and handed back nine findings nobody had checked, and 61 cents once they were.
• Both Anthropic flagships refused the patch task, both Geminis passed it, and GPT-5.6 Terra Pro's patch would not apply. The header fault was the one six of seven cheap models made on 2026-09-08.
• Reasoning tokens set the bill, not the listing. google/gemini-3.8-flash, listed at a fifth of Gemini 3.1 Pro's price, billed the same 24 cents for one regex change after 62K reasoning tokens.

Editor's notes

I wanted to do this special edition just to compare the free/cheap models directly to just running with frontier models, to check the hypothesis that it might be more expensive to check a worse model than to just use a better model to start with.

The moral of the story seems to be that you should always have something checking your code reviewer's findings, even if it's a really capable model. Dare I say that humans might still need to do code reviews after all? In any case, cheap models are still cheap even accounting for the cost of checking them, so that's an interesting result.

--Curtis

What's new this week

• anthropic/claude-opus-5 refused the patch fixture through OpenRouter on 2026-09-10, in three seconds, with nothing billed. The response carried finish_reason content_filter and a usage-policy message about "violative cyber content", the same refusal anthropic/claude-fable-5.1 gave the same bytes twice on 2026-09-06. The task asks for a regex change to a secret scanner and spells every credential as a placeholder. Why it matters: A request that touches secret-detection code can come back empty from either Anthropic flagship, so a pipeline built on them needs a fallback for that class of task. Claude Opus 5 on OpenRouter · the survey's 2026-09-10 runs · the record: the runs

• openai/gpt-5.6-terra-pro billed its prompt at three times the payload on all three runs. A 39 KB file counted 41,411 prompt tokens where Gemini counted 10,945 and Opus 5 counted 14,645, with 18,942 of them marked cached and 9,471 as cache writes. The 6 KB file counted 28,274. The bill followed the count. Why it matters: An input budget for a "pro" tier worked out from the listing and the file size will be low by about a factor of three. GPT-5.6 Terra Pro on OpenRouter · the survey's 2026-09-10 runs · the record: the runs

• google/gemini-3.1-pro-preview returned HTTP 504 "Upstream idle timeout exceeded" at 180 seconds on the patch fixture, with 1,908 completion tokens in flight and nothing billed. The same payload re-sent four minutes later answered in 152 seconds and passed. Why it matters: A first-party route on a frontier model can stall the way a shared free pool does, and the router gives up at three minutes of silence, so a script has to read the choice's error field and plan a resend. Gemini 3.1 Pro on OpenRouter · the survey's 2026-09-10 runs · the record: the runs

• google/gemini-3.8-flash spent 40,231 reasoning tokens on a 6 KB review and 61,867 on one regex change. At $3.75 per million output tokens that is 15 and 24 cents, against a listing that reads as a fifth of Gemini 3.1 Pro's. Gemini 3.1 Pro spent 11,669 and 20,117 on the same two tasks and billed 15 and 25 cents. Why it matters: Price a reasoning model by what it reasons, not by the per-token rate, or set a completion cap you can afford. Gemini 3.8 Flash on OpenRouter · the survey's 2026-09-10 runs · the record: the runs

Top things we learned

• Fourteen of the 18 findings the four frontier models produced on the zero-defect control were wrong, and one more was true but harmless. The file carries two known defects. Opus 5 found the network defect, named the stat defect with a reason the source does not support, and added six more that do not hold at the pin. GPT-5.6 Terra Pro found one and added four. Gemini 3.8 Flash found one and added one. Gemini 3.1 Pro found neither and added two, both findings the record already holds as inventions from cheaper models. Why it matters: The expensive model's findings list still has to be read against the source, so the reading is a cost of every arm and not a cost of the cheap one. the survey's 2026-09-10 runs · the record: the arms · the record: the runs

• A cheap model with an Opus 5 check cost less than a frontier model on its own, and the cheap one arrived checked. On the same 6 KB file, z-ai/glm-5.3-flash plus a metered Opus 5 check came to 16 cents for one real finding, and deepseek/deepseek-v4-flash plus the same check came to 23 cents for one. Opus 5 alone billed 24 cents, Gemini 3.1 Pro 15 cents, and GPT-5.6 Terra Pro 31 cents, and none of those figures includes reading the output. With an Opus 5 check added, the four frontier runs came to 32, 35, 61, and 72 cents. Why it matters: The checking step is the bill either way, so what sits in front of it should cost a cent, not a quarter, for the same verified yield on this file. the record: the arms · the record: the runs

• Ask for findings rather than a patch, at every price. GPT-5.6 Terra Pro's diff had a hunk header one line short, the fault that six of seven cheap models made on 2026-09-08, with all eight pattern verdicts right behind it. Gemini 3.1 Pro and Gemini 3.8 Flash each produced a patch that applied and passed every gate. Opus 5 refused the task. Across the record, five paid models have passed this fixture as delivered, and three of them cost under 16 cents. Why it matters: A frontier price does not buy a header that applies, so keep git apply --recount in the loop or write the patch from the findings yourself. the record: the runs

• Only Fable 5.1 has found both defects on the control in one run, and it brought an invention with it. Fable 5.1 hedged both as uncertain on 2026-09-06 and added one wrong finding. Opus 5 named both, but its stat-defect finding rested on a reason the pinned source does not support, and a finding is scored as written, so it counts as one real and seven wrong. Gemini 3.7 Flash and Gemini 3.8 Flash each found the stat defect alone, with zero and one invention. GLM-5.3 Flash, GPT-5.6 sol, and GPT-5.6 Terra Pro each found the network defect alone. Why it matters: One run from one model, at any price, has missed at least one of the two defects in every case but one, and that one cost 37 cents plus the reading of five findings. the record: the arms · the record: the runs

Models we're using this week

The manifest is unchanged from issue 2, which says why each model is in it. A frontier model run as a reference never enters the manifest, whatever it scores.

What a review cost

Every figure here is what OpenRouter billed, at list price through the API. The checking model is named beside every checking figure. Opus 5 is priced at $5 in and $25 out per million and Fable 5.1 at $10 in and $50 out. A reader whose checking model is covered by a subscription pays in quota rather than dollars, and the two are not the same bill.

On the 6 KB zero-defect control, the model half of a cheap run is a fraction of a cent and the checking half is the whole bill. GLM-5.3 Flash's run was $0.0015 and its Opus 5 check was 16 cents. DeepSeek V4 Flash's run was $0.0016 and its Opus 5 check was 23 cents. The frontier runs cost 15 to 31 cents before anyone read them, and their Opus 5 checks cost 18 to 41 cents on top: Gemini 3.1 Pro 32 cents in all, Gemini 3.8 Flash 35 cents, Opus 5 61 cents, GPT-5.6 Terra Pro 72 cents. Under Fable 5.1 as the checker every check cost more, from 34 to 62 cents, and the four frontier totals ran from 49 to 85 cents.

The checking half did not shrink with the price of the model being checked. Opus 5's nine findings cost 38 cents for Opus 5 itself to check, more than the 24-cent run that produced them. Gemini 3.1 Pro's two findings, both wrong, cost 18 cents to check.

A metered check is one request that reads the source and cannot reproduce anything. It is the fallback, and its price is the price of the fallback. The full tables, one tab per checking model, are in the Oxbox Survey costs workbook, with the same-batch, per-fixture, ratings, and rubric tabs.

How far to trust this

Everything above is what the survey tried, not what a vendor said. Three tiers, once: measured means the catalog capture, observed means a run or a probe this survey made, and reported means a vendor or a third party said it.

Each frontier model ran each fixture once, on 2026-09-10, through OpenRouter. One run is one observation, and it is not a property of the model. The zero-defect control is a single 6,096-byte Python file whose two known defects were found by earlier runs, so a model's count on it says how much it invents on a small file, not how much it finds on a large one. The grounding quiz is ten questions about one file. The patch fixture is one regex change.

Two checking models read every finding on the control, and they disagreed on 3 of the 18 verdicts. Where they split, the record went with the verdict that held to the pinned source, and the runs in the record say which. Both checks read the code and ran nothing.

A cost here is the model's bill plus a checker's, at list price through the API. A reader on a subscription pays in quota, not dollars.

Free means the vendor may train on what you send. Everything here is public code, so that is priced in and it is not a reason a model is excluded.

A model's vendor can be on a United States restricted-party list and nothing in a catalog listing says so. The lists and this edition's vendors are set out in the record, under Restricted-party listings.

The scores are measured from the record. The Editor's Rating is one person's judgment, said as such.

---

The record

The arms

Every model below ran the same three fixtures through oxbox on OpenRouter. The zero-defect control is a 6,096-byte Python file with two known defects at its pin, so a run's count reads as findings emitted against findings real. Prices are the listing in and out per million tokens on the day of the run. The last column is the run's bill plus a metered Opus 5 check of its findings where one was made, and the run's bill alone where the fixture is scored by a script.

Two models in the table gave the ideal answer on the control, zero findings with reasons: MiMo v2.5 and Dots 3 Note Preview. nvidia/nemotron-3.5-lightning did the same in the 2026-09-08 batch. A zero-finding run has nothing to check, so its total is its run.

On the grounding quiz, every model in the table that answered scored 10 of 10 except Mistral Small at 9. The four frontier runs billed $0.14 for Opus 5, $0.08 for GPT-5.6 Terra Pro, $0.05 for Gemini 3.1 Pro, and $0.02 for Gemini 3.8 Flash. DeepSeek V4 Flash and GLM-5.3 Flash billed $0.001 each, and the three free models billed nothing.

On the patch fixture, applied as delivered and passed every gate: Gemini 3.1 Pro ($0.25), Gemini 3.8 Flash ($0.24), Gemini 3.7 Flash ($0.15), GPT-5.6 sol ($0.04), and GLM-5.3 Flash ($0.006). Header would not apply, eight of eight verdicts behind it: GPT-5.6 Terra Pro ($0.13), Sonnet 5, DeepSeek V4 Flash, MiMo v2.5, Nemotron 3.5 Lightning, and North Mini Code. Refused: Opus 5 and Fable 5.1.

The runs

Every run below went through oxbox on OpenRouter on 2026-09-10, unpinned, effort high, a 100,000-token completion budget, temperature 0.2. The fixtures are the ones issue 2 described: the zero-defect control, the grounding quiz, and the patch fixture. Two checking models read every finding on the control as one metered request each, with the same verifier instructions and the same five pinned files.

anthropic/claude-opus-5, three runs, a reference and never a manifest entry.

• Grounding quiz: 10 of 10 in 37 seconds, $0.1435, route Claude Platform on AWS. It declined all three unanswerable questions and cited the timeout constant as ox's own patience rather than the provider's latency.
• Zero-defect control: 9 findings, 1 real, 1 benign, 7 invented, 118 seconds, $0.2373. The real finding is the network defect, asserted without a hedge. Its first finding named the stat defect but gave the wrong reason for the state that triggers it, and both checkers refuted it as written; a finding is scored as written, so it counts as invented. The two findings it hedged as uncertain were both standing inventions.
• Check by Opus 5, one metered request: $0.3764, 143 seconds, 3 confirmed and 6 refuted. Check by Fable 5.1 the same way: $0.6152, 109 seconds, 2 confirmed and 7 refuted. The two disagreed on one finding of the nine.
• Patch fixture: refused, content_filter, three seconds, nothing billed.

google/gemini-3.1-pro-preview, three runs and one stalled request, a reference.

• Grounding quiz: 10 of 10 in 20 seconds, $0.0543, ten one-line answers.
• Zero-defect control: 2 findings, 0 real, 94 seconds, $0.1474, 11,669 reasoning tokens. The first is the directory-detection claim three other models have made and the second, hedged as uncertain, is the UDP packet-filter claim two others have made. Both are standing inventions for this file.
• Check by Opus 5: $0.1762, 52 seconds, both refuted. Check by Fable 5.1: $0.3858, 61 seconds, both refuted.
• Patch fixture, first attempt: HTTP 504 "Upstream idle timeout exceeded" at 180 seconds, 1,908 completion tokens consumed, cost 0. Second attempt: applied as delivered, 8 of 8 verdicts, zero self-hits, 152 seconds, $0.2531, 20,117 reasoning tokens.

google/gemini-3.8-flash, three runs, a reference. Listed at the same price as Gemini 3.7 Flash.

• Grounding quiz: 10 of 10 in 22 seconds, $0.0225, the only run of the four the scorer passed without a reader.
• Zero-defect control: 2 findings, 1 real, 1 invented, 237 seconds, $0.1541, 40,231 reasoning tokens. The real finding is the stat defect, with the correct cause. The invention is the empty-value blind spot in the environment check, which needs a launcher that passes the parent environment through, and this one does not.
• Check by Opus 5: $0.1978, 62 seconds, one confirmed and one refuted. Check by Fable 5.1: $0.3399, 45 seconds, the same.
• Patch fixture: applied as delivered, 8 of 8 verdicts, zero self-hits, 370 seconds, $0.2417, 61,867 reasoning tokens.

openai/gpt-5.6-terra-pro, three runs, a reference. Prompt tokens counted at about three times the payload on every run.

• Grounding quiz: 10 of 10 in 20 seconds, $0.0822, 41,411 prompt tokens with 18,942 cached.
• Zero-defect control: 5 findings, 1 real, 4 invented, 147 seconds, $0.3061, 28,274 prompt tokens. The real finding is the network defect. The four inventions need a launcher that drops a variable, a policy that grants metadata on one home path and not another, or a work directory holding a read-only file under the test's own name.
• Check by Opus 5: $0.4149, 167 seconds, one confirmed and four refuted. Check by Fable 5.1: $0.5023, 90 seconds, three confirmed and two refuted. The two disagreed on two of the five, and the record went with the verdicts that held to the pinned tree.
• Patch fixture: header one line short, "corrupt patch at line 15", applies with --recount, 8 of 8 verdicts behind it, zero self-hits, 59 seconds, $0.1326.

The cheap and free rows in the arms table are the runs issues 1 and 2 reported, on the same three fixtures: the 2026-09-02 reference runs of Sonnet 5, Gemini 3.7 Flash, GPT-5.6 sol, DeepSeek V4 Flash, and GLM-5.3 Flash, the 2026-09-06 Fable 5.1 reference runs, the 2026-09-06 and 2026-09-07 candidate runs of DeepSeek V4 Flash and GLM-5.3 Flash, and the 2026-09-08 batch of seven cheap and free models. The Opus 5 checks quoted for GLM-5.3 Flash, DeepSeek V4 Flash, Mistral Small, and gpt-oss-120b are the metered checks issue 2 reported. The DeepSeek row is the 2026-09-06 re-send that answered in 47 seconds, not the run that returned nothing after 33 minutes.

Restricted-party listings

A model's vendor can be on a United States restricted-party list and nothing in a catalog listing says so. What follows is stated as facts about lists, never as legal advice. A reader in a regulated setting should check the lists themselves and their own procurement rules.

Of the vendors in this edition, Zhipu AI, behind z-ai/glm-5.3-flash, was added to the United States Commerce Department's Entity List effective 2025-01-16, per the Federal Register, 2025-00704. Being listed does not by itself forbid a customer from calling the vendor's hosted API, but procurement and compliance policies commonly key on the list. inclusionAI, behind inclusionai/ling-3.0-flash-fin:free, is Ant Group's open source arm, and Ant Group is an Alibaba affiliate. Alibaba was added to the Defense Department's 1260H list in June 2026, per WilmerHale, 2026-06-11. Neither Ant Group nor inclusionAI appears in that citation, so the listing does not inherit the flag. Anthropic, Google, OpenAI, Mistral, DeepSeek, Xiaomi, Cohere, and dots studio appear in neither citation. The record is the two citations, not a fresh check of the lists.

Sources

Fetched and used:

• OpenRouter chat completions: every run and every metered check in this edition, on 2026-09-10, with the billed cost read from each response's usage block.
• OpenRouter models API: the listing prices in the arms table, captured 2026-09-10.
• Claude Opus 5 on OpenRouter, Gemini 3.1 Pro on OpenRouter, Gemini 3.8 Flash on OpenRouter, and GPT-5.6 Terra Pro on OpenRouter: the four frontier listings.
• Federal Register, 2025-00704: the Entity List addition of Zhipu AI.
• WilmerHale, 2026-06-11: the June 2026 1260H update.
• oxbox: the tool every run went through, version 1.3.0.
• Oxbox Survey costs: the full cost tables, one tab per checking model.
• Issue 2, 2026-09-10: the cheap and free runs this edition compares against.

Not fetched this edition: the catalog snapshot, the access probe, and the stealth page. This is a comparison on runs already in the record plus four new reference models, and the catalog is issue 2's until the next weekly.

---

This edition is published from its source file. The evidence behind it, the catalog snapshots and the manifests, is in oxbox-survey, and nothing from there is edited on the way here.