We publish the rows we lose, too.

Tools not shown. CodeRabbit's terms bar publishing benchmark results about it without consent, Cursor's Bugbot terms allow it only alongside everything needed to replicate the test, and we removed Greptile's results because its terms restrict sharing the service's results with third parties without written authorization. We would add any of them if they agree.

Full method, limitations and the reproduce script, Experiment 8 →

DatasetHarnessPaper (arXiv 2603.26130)

*Competitor costs are priced at Aletheore's bare-mode token volume for the same 100 tasks; per-file mode makes more calls, which is why our own figure is higher there.

Method, every caveat and the per-task data →

Read honestly, not cherry-picked: run 3's apparent 100% is not evidence that full context caught up. A transient network failure happened to strip out exactly the harder cases that produced its rejects and uncertains in the other two runs. Evidence-only never underperformed full context in any run.

How much smaller is the prompt, actually?

Read the last row honestly rather than expecting a 6 to 10 times price drop to match the token drop: on this model, completion tokens run 11k to 13k almost regardless of which arm generated the prompt, so the real dollar savings land at roughly a quarter, not a multiple. What the input-side reduction buys is headroom against a tight tokens-per-minute quota, such as a free-tier provider's 6,000 TPM.

Result: evidence-only shipped as Flash Review's production default, not an experiment behind a flag. Total cost for all three validation runs on the real production model: $0.9229.

5-0-2

wins, losses, ties on top-1

all 7 shared corpora

$0.00

to build a searchable index

against RepoWise's $1.85

93.3%

cross-language top-5 accuracy

up from 60.0% after a language pre-filter fix

Top-1: 5 wins, 0 losses, 2 ties. Flask (measured separately, not one of the seven shared corpora): Aletheore 68.8% / 93.8% / 100% (top-1/3/5) against RepoWise semantic 28.1% / 56.2% / 56.2%. Where we lose: jekyll top-5, 46.7% against their 66.7%.

Both systems were also given vocabulary-phrased questions on the same wikis. RepoWise gains from this too, because its wiki pages name the symbols, and on jekyll it overtakes us outright: 80.0% to 66.7%. Across those ten cells we lead in seven, tie in two and lose one. RepoWise's --mode symbol returned 0.0% on every corpus and is excluded as a misapplied mode, not counted as a loss.

Search latency, measured in-process for both tools: Aletheore 40.5 ms mean against RepoWise's 52.5 ms, reversed from an earlier 125 ms against 68 ms figure (see METHODOLOGY.md).

• Aletheore ties or leads Graphify on every question once a ground-truth error found in pre-publication review was corrected.
• Excluding the one question both tools timed out on, Aletheore answers for 36% fewer tokens than Graphify (9,803 against 15,296 mean).
• The honest caveat: only 2 of 15 questions actually discriminate between conditions, so this is a real result on this question set, not a claim that generalizes past 15 questions on one corpus. Total real cost of the whole comparison: $0.19.

Method, both corrections made before publishing, and the setup-time comparison →

We lead on average, not decisively. Most individual per-corpus gaps sit inside that corpus's own measured judge-repeat spread. The aggregate preference count across all 360 judged pairs is more telling: Aletheore 198 (55.0%), RepoWise 144 (40.0%), tie 18 (5.0%). This reverses an earlier loss (2.13 against 2.35) measured on different code with a different method.

The full history, including a fix that shipped and sat undetected for months →

Methodology and known limitations →

Each corpus's full case tables and root-cause write-ups →

Why Qwen3-Embedding-8B: level on quality, much faster

We tested nine configurations (eight hosted models and one query-instruction variant) against the production Jina model on the same code-search questions as the retrieval matrix: 257 questions used to choose, 90 held out. Qwen3-Embedding-8B is level with Jina on retrieval quality, not better: +0.008 MRR with a 95% interval of -0.019 to +0.035 on 332 questions for the shipped setting, and the same on the held-out questions. Asking for a 1024-dimension vector scores the same as the native 4096. Three of the other models (bge-m3, bge-base, Nemotron 1B) were clearly worse than Jina. What changed is operational: on one production host, bulk indexing ran about 7 times faster with one request at a time and up to about 17 times faster with four in parallel, and the CPU-bound sidecar can be retired. Embedding code now means sending code chunks to the provider; the privacy policy names it and the CLI says so before it uploads.

Small effects on a small set: a 95% interval on MRR is about plus or minus 0.03, so differences under that are noise. The speed figures are for one host's CPU and vary with provider load; the raw runs, including one that did not scale, are published with the method and limitations.

Full method, limitations and the reproduce script →

Earlier measurement: local nomic against hosted jina

Mean: top-1 +5.0 points, MRR +0.049. 20 of 23 rows flat or better; 3 worse, all in Slim's vocabulary regime or zod. Thrift's cross-language row is the largest single gain (+20.0 points), the same regime the language pre-filter fix targeted.

Update, 2026-08-20: a later, independent hosted-jina measurement against the live service found a much steeper zod gap than the -6.7 points above (general top-1 20.0% to 0.0%, vocabulary 60.0% to 6.7%). Two decoy files that jina ranks above the true answer explain it: a 37-line smoke-test file that imports every zod build variant, and the roughly 30 locale files under packages/zod/src/.

Every coding agent that reads this file pays less to do it. Measured with the o200k_base tokenizer on a fresh self-scan of this repository (2026-09-25 snapshot; 5,112,929 against 2,362,099 bytes). The absolute counts change as the repo grows.

• !Five of eight corpora score below 35% top-1 under vocabulary-avoiding phrasing, though most of that gap is our question authoring, not the product. Every one recovers 20 to 47 points when asked in the project's own terms.
• !AIRview writes a page for only 21 of 100 changed files on Flask's last 30 commits; the rest are served by a deterministic fallback, not the generated wiki this project is named for.
• !"How does X work?" comprehension is close, not a clean win: 2.00 against 1.77 averaged across five languages, but most per-corpus gaps sit inside the judge's own noise.
• !On SWE-PRBench our absolute recall (22.3%) and precision (20.5%) are low, as they are for every model; the score is above the published cluster by a margin close to our own run-to-run variance.
• !The Graphify comparison rests on 15 questions on one corpus, of which only 2 discriminate between conditions.
• !Our local setup was slower than Graphify's: scanning and indexing ERPNext took about 23 minutes against their about 1 minute. We found the real bottleneck (dead-code detection, 77% of scan time, not parsing) and fixed it in aletheore 0.9.5: 53.23 s against the old 236.02 s, verified against the installed release. Indexing (about 19 minutes, a separate I/O-bound step) is now the honest remaining gap.
• !The PR-review comparison rests on 13 pull requests, so its intervals are wide and small differences between tools are noise. Our fix for cross-file false positives was developed while looking at these same pull requests and has not been checked on unseen ones. "Precision" counts any accurate, specific claim about the diff, so it measures accuracy of claims, not how important they are. The head-to-head runs the diff-only generation path; a separate four-run measurement of the production call scored 89.2% precision, and it too was developed while looking at these pull requests. The other tools' rows are single runs from before 2026-09-23.