Evaluation

Each case runs the engine's fix path, the one the worker calls, without the worker's dependency-file updater, PR limits or per-repository backend choice. Teire fetches the repository, assesses it (registries, API diff, locate), runs the fix with each model as the agent, and verifies in a sandbox, the repository's own checks before and after: Cloudflare Containers for the first rounds, Token Factory Sandboxes for the four model retirements in Go, Java, Ruby and Rust. Every repository is public as a folder of teire-dev/examples, with tests that pass on the old version. For the model retirements, the tests fail because the old model is gone.

The columns read evidence · attempts · agent steps · model cost.

Package majorsCopy Anchor Link

Model retirementsCopy Anchor Link

The four language cases share one repository with the AI web case, so each run swaps the retired ID in all five folders and leaves the test stand-in that lists retired IDs alone. Each folder's tests call a local stand-in that answers like OpenAI's API: the retired ID gets the API's 404, so the tests fail before (0/2) and pass after (2/2). No agent was needed; Nemotron 3 Ultra read each diff.

Clean repositories: no false positivesCopy Anchor Link

No clean repository was called affected. Three read inconclusive because their searches work on syntax, and Teire says not affected only with type-level coverage.

Reading the diffsCopy Anchor Link

Verified is necessary, not sufficient, so every patch was also read.

• Super passed by bypassing the model. On demo-py-agent it removed temperature=0 and added a keyword rule: if any(word in ticket for word in ["charge", "invoice", …]): return "billing". The tests passed because the rule classified without the model. Ultra instead raised max_tokens to make room for the reasoning model's answer.
• Both widened a schema. On zod, both models wrote z.record(z.string(), z.unknown()) (or z.any()), where the faithful zod 4 form is z.record(z.string(), z.string()). No test caught it.

Teire added a review step for this. After a fix passes, Nemotron reads the final diff against the quoted notes and the rules. It flags changed behaviour, widened validators, weakened tests and ignored replacements, and sends them back to the agent once. On the rerun, the review caught Ultra's widened zod schema. The next attempt wrote z.record(z.string(), z.string()) and verified. Review points never decide ready or draft: evidence does.

What the evaluation fixedCopy Anchor Link

Each bug below was found by a failing case, fixed, and the case rerun:

The pass barCopy Anchor Link

The plan set the bar as "Ultra ≥ Super on fixes, and the false-positive rate is recorded".

• Ultra: every fix verified; on reading, its only flaw was the zod widening both models shared, which the review step now catches.
• Super: every fix verified, one rejected on reading.
• False positives: none, across six clean repositories.

Super costs 3 to 7 times less per run and was the failover.

The failover since 2026-10-30: Nemotron 3.5 SuperCopy Anchor Link

Token Factory retires Nemotron 3 Super on 2026-11-09 and names nvidia/Nemotron-3.5-Super-VL as its replacement, at the same price. Teire switched on 2026-10-30. The twelve cases above were not rerun on it. It was measured on the model-comparison jobs (scripts/compare-models.ts), with the same prompts and settings as production (reasoning off for extraction):

3.5 Super writes fixes as well as Ultra, but often paraphrases its quotes when it reads. It reads only while Ultra is down; then Teire's quote gate holds back every record whose quote isn't on the page, so a paraphrase is never stored as fact.