Benchmarked, not just claimed
We publish the rows we lose, too.
A separate, public repository measures Aletheore head-to-head against RepoWise — locating code, generated docs, PR review quality, and what a bare LLM gets wrong that a deterministic scanner gets right. Every retrieval number below recomputes from raw results with no API key and no network; every PR-review number comes from real API runs with the actual production model.
git clone https://github.com/Aletheore/aletheore-benchmarks
cd aletheore-benchmarks
python scripts/score_fallback_judge.pyLocating code, head-to-head against RepoWise
all 7 shared corpora
vs. RepoWise's $1.85
accuracy
Locating code · same questions, same ground truth
Every shared corpus, both systems, best RepoWise mode.
RepoWise searches its own generated wiki, so each corpus had one built for it first
(init --coverage 1.0, deepseek-v4-flash, 1,341 pages, $1.85 total). The
table below shows RepoWise's best of semantic or full-text search per corpus, so
it isn't beating a weaker mode by default.
Locating code, top-1: Aletheore vs. RepoWise (best mode)
Same questions, same ground truth, all 7 shared corpora
| Corpus | Language | Aletheore | RepoWise semantic | RepoWise full-text | Winner |
|---|---|---|---|---|---|
| gin | Go | 80.0% | 60.0% | 46.7% | ✅ Aletheore |
| serde | Rust | 53.3% | 6.7% | 13.3% | ✅ Aletheore |
| gson | Java | 40.0% | 26.7% | 0.0% | ✅ Aletheore |
| jekyll | Ruby | 26.7% | 13.3% | 6.7% | ✅ Aletheore |
| Slim | PHP | 26.7% | 26.7% | 20.0% | 🟰 tie |
| guzzle | PHP | 20.0% | 13.3% | 20.0% | 🟰 tie |
| zod | TypeScript | 20.0% | 6.7% | 13.3% | ✅ Aletheore |
Top-1: 5 wins, 0 losses, 2 ties. Our weakest languages still match or beat them. Flask (measured separately, not one of the seven shared corpora above): Aletheore 68.8% / 93.8% / 100% (top-1/3/5) against RepoWise semantic 28.1% / 56.2% / 56.2%. Where we lose: jekyll top-5, 46.7% against their 66.7% — see "Where we lose" below.
Both systems were also given vocabulary-phrased questions on the same wikis. RepoWise gains from this too — its wiki pages name the symbols — and on jekyll it overtakes us outright:
| Corpus | Aletheore general | RepoWise general | Aletheore vocabulary | RepoWise vocabulary |
|---|---|---|---|---|
| Slim | 26.7% | 26.7% | 73.3% | 66.7% |
| gson | 40.0% | 26.7% | 60.0% | 46.7% |
| zod | 20.0% | 13.3% | 60.0% | 26.7% |
| guzzle | 20.0% | 20.0% | 53.3% | 53.3% |
| jekyll | 26.7% | 13.3% | 66.7% | 80.0% |
Across those ten cells we lead in seven, tie in two, and lose one
— jekyll's vocabulary regime, where RepoWise's wiki naming its own symbols overtakes
us 80.0% to 66.7%. One thing this deliberately doesn't claim: RepoWise's
--mode symbol returned 0.0% on every corpus and is excluded as a
misapplied mode, not counted as a loss. Search latency, measured in-process for both
tools (previously only Aletheore's had a committed script for this — RepoWise's own
number was CLI-subprocess time, which pays a real but unrelated Python-import cost
each call): Aletheore 40.5ms mean against RepoWise's 52.5ms, reversed from an earlier
125ms-vs-68ms figure — see
METHODOLOGY.md.
| Aletheore | RepoWise | |
|---|---|---|
| Indexing cost, 7 corpora | $0.00 | $1.85 |
| Per corpus | $0.00 | $0.09 – $0.47 |
| What it costs money for | nothing — local nomic-embed-text | LLM generation of 1,341 wiki pages |
| Typical setup time | seconds to ~1 min per corpus | minutes per corpus |
Full retrieval matrix — top-1 / top-3 / top-5 / MRR, all 12 corpora, 23 regime rows
Aletheore 0.8.11 installed from PyPI, each corpus re-scanned and re-indexed from
scratch with local nomic-embed-text (768-dim) embeddings, no API key.
Two phrasing regimes are published per corpus where available: general
deliberately avoids the project's own vocabulary, vocabulary uses it — real
users ask somewhere between the two, so a language's true figure is bracketed by
both rather than given by either. Every weak corpus moves 20-47 points on phrasing
alone, larger than any ranking change in this programme.
Top-1, general phrasing, by corpus — the spread the average hides
Go and Python are strong; Java, Ruby, PHP, and TypeScript are not — and the weak corpora were measured last
| Corpus | Regime | Top-1 | Top-3 | Top-5 | MRR | n |
|---|---|---|---|---|---|---|
| location (Flask) | general | 71.9% | 93.8% | 100.0% | 0.832 | 32 |
| gin | general | 80.0% | 100.0% | 100.0% | 0.878 | 15 |
| serde | general | 53.3% | 66.7% | 73.3% | 0.617 | 15 |
| Slim | general | 26.7% | 60.0% | 66.7% | 0.458 | 15 |
| Slim | vocabulary | 73.3% | 80.0% | 93.3% | 0.797 | 15 |
| guzzle | general | 20.0% | 53.3% | 66.7% | 0.374 | 15 |
| guzzle | vocabulary | 53.3% | 93.3% | 100.0% | 0.728 | 15 |
| jekyll | general | 26.7% | 33.3% | 46.7% | 0.337 | 15 |
| jekyll | vocabulary | 66.7% | 86.7% | 93.3% | 0.778 | 15 |
| zod | general | 20.0% | 40.0% | 40.0% | 0.289 | 15 |
| zod | vocabulary | 60.0% | 73.3% | 73.3% | 0.667 | 15 |
| gson | general | 40.0% | 66.7% | 80.0% | 0.544 | 15 |
| gson | vocabulary | 60.0% | 86.7% | 86.7% | 0.749 | 15 |
| axios | general | 20.0% | 46.7% | 66.7% | 0.407 | 15 |
| axios | vocabulary | 73.3% | 93.3% | 100.0% | 0.850 | 15 |
| jq | general | 53.3% | 66.7% | 80.0% | 0.648 | 15 |
| jq | vocabulary | 73.3% | 100.0% | 100.0% | 0.867 | 15 |
| fmt | general | 40.0% | 60.0% | 86.7% | 0.527 | 15 |
| fmt | vocabulary | 66.7% | 93.3% | 93.3% | 0.789 | 15 |
| AutoMapper | general | 6.7% | 20.0% | 33.3% | 0.177 | 15 |
| AutoMapper | vocabulary | 86.7% | 100.0% | 100.0% | 0.933 | 15 |
| thrift | general | 6.7% | 33.3% | 53.3% | 0.241 | 15 |
| thrift | cross-language | 53.3% | 73.3% | 93.3% | 0.677 | 15 |
All 11 supported languages are measured here, across 12 single-language corpora
plus apache/thrift — the first genuinely polyglot corpus (eight
languages, none above a third of the modules), which implements the same protocol
separately in each language and tests whether retrieval can tell one language's
implementation from another's. A 0.8.10→0.8.11 fix added a language
pre-filter that raised thrift's cross-language top-5 from 60.0% to 93.3% — the one
defect found here that good phrasing does not explain, since the cross-language
questions already use full vocabulary. Full methodology, including two ranking
fixes that were built and rejected on measurement, in
METHODOLOGY.md.
Hosted embeddings: jina vs. local nomic — full 23-row comparison
This one doesn't carry the same "no API key, no network" guarantee as the
rest of this page — it measures Aletheore's hosted embedding endpoint
(jina-embeddings-v2-base-code, requires aletheore login and
a paid plan) against the same 13 corpora, 23 corpus/regime pairs, and questions as
the table above, with only the embedder changed from local
nomic-embed-text (768-dim) to hosted jina (768-dim).
| Corpus | nomic top-1 | jina top-1 | Δ top-1 | nomic MRR | jina MRR |
|---|---|---|---|---|---|
| flask (location) | 71.9% | 81.2% | +9.3pp | 0.832 | 0.901 |
| gin | 80.0% | 86.7% | +6.7pp | 0.878 | 0.933 |
| serde | 53.3% | 53.3% | +0.0pp | 0.617 | 0.678 |
| Slim (general) | 26.7% | 53.3% | +26.6pp | 0.458 | 0.647 |
| Slim (vocab) | 73.3% | 66.7% | -6.6pp | 0.797 | 0.811 |
| guzzle (general) | 20.0% | 20.0% | +0.0pp | 0.374 | 0.458 |
| guzzle (vocab) | 53.3% | 73.3% | +20.0pp | 0.728 | 0.844 |
| jekyll (general) | 26.7% | 26.7% | +0.0pp | 0.337 | 0.382 |
| jekyll (vocab) | 66.7% | 66.7% | +0.0pp | 0.778 | 0.791 |
| zod (general) | 20.0% | 13.3% | -6.7pp | 0.289 | 0.237 |
| zod (vocab) | 60.0% | 53.3% | -6.7pp | 0.667 | 0.658 |
| gson (general) | 40.0% | 46.7% | +6.7pp | 0.544 | 0.569 |
| gson (vocab) | 60.0% | 66.7% | +6.7pp | 0.749 | 0.797 |
| axios (general) | 20.0% | 20.0% | +0.0pp | 0.407 | 0.420 |
| axios (vocab) | 73.3% | 80.0% | +6.7pp | 0.850 | 0.878 |
| jq (general) | 53.3% | 53.3% | +0.0pp | 0.648 | 0.728 |
| jq (vocab) | 73.3% | 80.0% | +6.7pp | 0.867 | 0.883 |
| fmt (general) | 40.0% | 46.7% | +6.7pp | 0.527 | 0.621 |
| fmt (vocab) | 66.7% | 66.7% | +0.0pp | 0.789 | 0.789 |
| AutoMapper (general) | 6.7% | 13.3% | +6.6pp | 0.177 | 0.243 |
| AutoMapper (vocab) | 86.7% | 86.7% | +0.0pp | 0.933 | 0.922 |
| thrift (general) | 6.7% | 20.0% | +13.3pp | 0.241 | 0.285 |
| thrift (cross-language) | 53.3% | 73.3% | +20.0pp | 0.677 | 0.822 |
Mean: top-1 +5.0pp, MRR +0.049. 20 of 23 rows flat or better; 3 worse, all
in either Slim's vocabulary regime or zod. Thrift's cross-language row is
the largest single gain — the same regime the language pre-filter fix targeted, so
hosted jina extends that gain rather than eroding it. Checked directly rather than
left as a footnote: zod's regression traces to near-duplicate crowding between its
parallel classic and mini API variants, not a scanner bug
— both embedders hit the same 6/15 top-5 rate on zod's general questions, they just
disagree on which two they miss.
Update, 2026-08-20: a later, independent hosted-jina
measurement against today's live service (not this section's dev checkout) found a much
steeper zod gap than the -6.7pp above — general top-1 20.0% → 0.0%, vocabulary 60.0% →
6.7%. Reading the raw ranked files (not just the score) found two concrete decoy files
jina ranks above the true answer that nomic does not: packages/resolution/src/index.ts,
a 37-line smoke-test file that literally imports every zod build variant in one place
(zod, zod/mini, zod/v3, zod/v4,
zod/v4-mini, plus a locale file), giving it lexical overlap with nearly the
entire module surface despite being semantically irrelevant to any single one of them —
it shows up as a false top-3 hit on 4 of 15 general questions; and the ~30 locale files
under packages/zod/src/v4/locales/, each importing the same core types
($ZodStringFormats, errors, util) as the real
implementation files they're competing against, because every locale wires up the same
error-message keys. Neither is a broken pipeline — both are plausible, on-topic, wrong
answers, the same near-duplicate-crowding category above, just a sharper instance of it.
Why this measurement's gap is so much larger than the -6.7pp already published above is
not yet resolved — the two were measured against different deployments, so infrastructure
drift between them hasn't been ruled out. Full account in
METHODOLOGY.md.
Answering from a file — fallback vs. RepoWise get_context:
blind pairwise judge, 0-3, three repeats, both systems' positions swapped each
grading. On the seven files where both return substantive material, Aletheore's
file fallback scores 2.857 against RepoWise's 2.000
— a +0.857 gap, identical in all three repeats. A further 15 files (YAML, TOML, RST,
lockfiles) are reported as coverage, not score: RepoWise returns "empty or non-symbol
file" or "Target not found" for all fifteen, which is it declaring those files out of
scope, not losing on quality — pooling those zeros into the headline would overstate
the gap, so they're kept separate instead.
PR review · shipped, not hypothetical
Dropping the raw file dump didn't cost review quality.
Flash Review can build its prompt with the full raw content of every changed file,
or with Aletheore's own evidence alone (blast radius, referenced symbols — no file
dump). Four experiments across three models asked whether that trade-off actually
costs anything. The one that decided it: gpt-5.6-luna, the real
production model, generating reviews under both arms; deepseek-v4-flash
independently verifying every finding against the diff, three full repeats of a
50-case mixed-language corpus.
| Run | Evidence only (compact) | + full file content |
|---|---|---|
| Run 1 | 97.7% verified-accept (42/43) | 85.7% (36/42) |
| Run 2 | 97.6% verified-accept (40/41) | 90.5% (38/42) |
| Run 3 | 96.7% verified-accept (29/30) | 100% (27/27) — coverage artifact* |
*Read honestly, not cherry-picked: run 3's apparent 100% isn't evidence full context caught up — a transient network failure happened to strip out exactly the harder cases that produced its rejects and uncertains in the other two runs. Evidence-only never underperformed full context in any run. Full writeup, including an earlier weaker-model run where the compact approach's real recall win came with a real cost — the highest false-positive rate of the arms tested — in pr_review/README.md.
How much smaller is the prompt, actually?
| Evidence only (compact) | + full file content | Compact is | |
|---|---|---|---|
| Median context size (measured) | 1,480 chars (~370 tok) | 15,212 chars (~3,800 tok) | ~10.3x smaller |
| Avg prompt tokens, real API (DeepSeek V4 Flash) | 2,483 | 16,325 | ~6.6x smaller |
| Avg cost per review, same run | $0.0041 | $0.0054 | ~24% cheaper |
Read the last row honestly rather than expecting a 6-10x price drop to match the token drop: on this model, completion tokens run 11k-13k almost regardless of which arm generated the prompt — reasoning-mode verbosity, a model-level trait unrelated to context strategy — so the real dollar savings land at roughly a quarter, not a multiple. What the ~6-10x input-side reduction actually buys is headroom against a tight tokens-per-minute quota (e.g. a free-tier provider's 6,000 TPM) — the difference between a review fitting in one call and not fitting at all.
Result: evidence-only shipped as Flash Review's actual production default, not an experiment behind a flag. Total cost for all three validation runs on the real production model, real API pricing: $0.9229.
Head-to-head against PR-Agent · a real, named competitor
Same model, same corpus, production code.
A different question from the compact-vs-context experiments above: how does
Aletheore's actual hosted product compare against a real, named, external competitor
(Qodo's PR-Agent), held
to the same model, so the result isolates review methodology rather than which vendor
has the bigger model budget? Both tools on gpt-5.6-luna, 24-case corpus,
production Aletheore deployed at commit 35e18f8.
vs. PR-Agent's 8
on identical footing
vs. PR-Agent (15.9s vs. 69.3s)
| Tool | Hit | Partial | Miss | False Positives |
|---|---|---|---|---|
| Aletheore AIR (Luna + DeepSeek verify) | 15 | 1 | 4 | 0 |
| Aletheore Flash (Luna only, no verify) | 15 | 0 | 5 | 0 |
| PR-Agent / Qodo (Luna) | 6 | 0 | 14 | 8 |
AIR and Flash tie on total recall (15/20 each) despite Flash never calling the verification model, consistent with verification being a precision mechanism, not a recall lever. Both Aletheore tiers hold a clean false-positive record on the 4 clean diffs, not explained by proposing fewer findings overall (it doesn't cost recall, see the hit numbers above). This reverses an earlier, unintentionally unfair run (free-tier Aletheore vs. PR-Agent's own pricier default model) that had shown the opposite, documented as superseded, not discarded. Disclosed limitations (no blind-judge pass this cycle, AIR measured via direct invocation rather than a live webhook) and the full setup in pr_review/README.md, Experiment 5.
Deterministic analysis vs. bare LLM
Can a bare LLM reproduce the scanner's answer, given the same data?
Not "Aletheore vs. RepoWise" — for the parts of Aletheore that aren't an LLM call at all (hotspots, ownership, dead code, computed from real git history and a real import graph), what happens if a model is simply handed the same data and asked the same question? On the flask corpus, given the exact git log slice and import statements the scanner itself consumes:
| Aletheore | Bare LLM (gpt-5.6-luna) | Bare LLM (gpt-5.6-terra) | |
|---|---|---|---|
| Hotspots (top 10) | exact, every run | declined — asked for real code instead | 0/10 correct, 4 fabricated |
| Ownership (top 8) | exact | 1/8 exact, drops a contributor | 4/8 exact, still fabricates a person |
| Dead code | 2/2, zero false positives | 48 flagged, 4.2% precision | 49 flagged, 4.1% precision |
Full methodology, including a real bug this testing surfaced, in DETERMINISTIC_VS_LLM.md.
Dead-code detection · Kotlin and Swift, vs. RepoWise
Swift: a clean win. Kotlin: an exact match.
A different axis from "how does X work?" above: which files does each tool correctly leave alone as real, reachable code? Measured file-level (RepoWise also flags unused-export symbols; Aletheore's dead-code module doesn't attempt that granularity, so symbol-level findings are excluded from both sides).
| Repo | Files | RepoWise false positives | Aletheore false positives |
|---|---|---|---|
| vapor/penny-bot (Swift) | 168 | 60 (36%) | 0 |
| vapor/api-template (Swift) | 10 | 6 (60%) | 0 |
| android/architecture-samples (Kotlin) | 268 | 7 | 7, same 7 files |
RepoWise's Swift support doesn't understand whole-module imports: a
Swift import Foo names a compiled target, not a file, so files referenced
only from outside Swift's import syntax (a Lambda handler invoked by the AWS runtime)
look completely unreachable to it. Getting Aletheore to 0 on Swift, and Kotlin from an
initial 31 down to an exact 7-vs-7 match with RepoWise, took five real fixes to
Aletheore's own scanner, found by re-running this comparison rather than trusting an
earlier pass: same-target and same-package implicit visibility, a manifest
string-interpolation parsing bug, and two missing import-resolution cases (top-level
Kotlin functions and constants). Full writeup and the fix history in
LANGUAGE_COVERAGE.md.
Head-to-head against Graphify · on their own primary-source corpus
We ran their benchmark, not just ours.
Graphify publishes
its own primary-source benchmark on ERPNext — a real ~1M-LOC Python
codebase, and the exact same corpus their own BENCHMARKS.md uses. Rather
than cite either tool's own numbers, we cloned ERPNext ourselves, ran both tools
ourselves under one shared agent loop and one anonymized judge, and published what
actually happened. A widely-circulated "70%/71.5x token reduction" figure attributed
to Graphify doesn't appear anywhere in their own repository — we don't use it here.
once a bad ground-truth fact we caught was fixed
the real, robust win
45 harness + 90 judge calls
| baseline (grep/read/list) | + Aletheore | + Graphify | |
|---|---|---|---|
| Coverage (30 samples) | 92.2% | 100.0% | 93.3% |
| Tokens/query, excl. one MAX_TURNS timeout | 9,157 | 9,803 | 15,296 |
A pre-publish review caught and fixed two real problems before this shipped: one question's ground truth was factually wrong (all three tools had answered it correctly and were wrongly capped), and the raw coverage gap was driven almost entirely by one question where baseline and Graphify both hit their turn limit without converging — a real, specific result, not a distributed lead. Only 2 of 15 questions discriminate between Aletheore and Graphify at all once the ground truth is fixed — honestly, that's a tie on quality. The number that holds up consistently, not as an artifact of one question, is token cost: Aletheore answers for 36% fewer tokens than Graphify. Full writeup, raw results, and the corrections themselves, in graphify_comparison/README.md.
Where we lose
Stated here rather than in a footnote.
⚠️ Five of eight corpora score below 35% top-1 under vocabulary-avoiding phrasing,
though most of that gap is our question authoring, not the product — every one
recovers 20-47 points when asked in the project's own terms.
⚠️ AIRview writes a page for only 21 of 100 changed files on Flask's last 30 commits;
the rest are served by a deterministic fallback, not the generated wiki this project
is named for.
⚠️ "How does X work?" comprehension is close, not a clean win — averaged across five
languages (Python, JavaScript, C#, C++, C) AIRview scores 2.00 against RepoWise's
1.77 on a blind 0-3 judge, but most individual per-corpus gaps sit inside that run's
own measured judge noise, so read this as "roughly at parity, leaning ahead," not a
decisive lead. This reverses an earlier 2.13-vs-2.35 loss, on different code and a
different measurement method — see
AIRVIEW_GAP.md
for the full history, including a real fix that shipped but sat undetected in the
benchmark for months.
⚠️ Our local setup was slower than Graphify's — scanning and indexing ERPNext took
~23 minutes against their ~1 minute. We found the real bottleneck (dead-code
detection, 77% of scan time, not parsing) and fixed it: shipped in aletheore 0.9.5,
verified live against the installed PyPI release at 53.23s vs the old 236.02s -
4.4x faster, real and confirmed. Indexing (~19min, a separate I/O-bound step) is
now the honest remaining gap, not scan.
Every number above is in a public repo, with the raw results.
418 questions across 12 corpora and 11 languages, a 50-case PR-review corpus, and the deterministic-vs-LLM writeup — all reproducible, all checked against real Aletheore output, not a mockup.