Benchmarks
8 repos, each pinned to a commit and labelled by hand following the labelling guide. The engine runs on every change in CI and the build fails if any category drops more than 2 points.
Latest run
| Category | Precision | Recall | F1 | Hallucination | TP / FP / FN |
|---|---|---|---|---|---|
| routes | 100.0% | 100.0% | 100.0% | 0.4% | 460 / 0 / 0 |
| models | 100.0% | 100.0% | 100.0% | 0.0% | 26 / 0 / 0 |
| services | 100.0% | 100.0% | 100.0% | 0.0% | 16 / 0 / 0 |
| edges | 100.0% | 100.0% | 100.0% | 0.0% | 168 / 0 / 0 |
F1 over time
Per repo
| Repo | routes F1 | models F1 | services F1 | edges F1 | Time |
|---|---|---|---|---|---|
| aadilsal/Munshi @ 40ad91b | 100.0% | 100.0% | 100.0% | 100.0% | 10.2s |
| aadilsal/MotorAi @ c10fa5d | 100.0% | 100.0% | 100.0% | 100.0% | 6.4s |
| aadilsal/AjaiConsultancyAssignment @ df83d72 | 100.0% | 100.0% | 100.0% | 100.0% | 10.5s |
| aadilsal/TournamentMatchMaker @ 973998e | 100.0% | 100.0% | 100.0% | 100.0% | 15.6s |
| aadilsal/MdcatExpert @ 34c226d | 100.0% | 100.0% | 100.0% | 100.0% | 12.8s |
| aadilsal/Insight-OS @ 38ded83 | 100.0% | 100.0% | 100.0% | 100.0% | 2.6s |
| aadilsal/SpotterTruckLogging @ 3cded29 | 100.0% | 100.0% | 100.0% | 100.0% | 4.2s |
| aadilsal/JobFinder @ 13d8d0d | 100.0% | 100.0% | 100.0% | 100.0% | 2.9s |
How to read this
- Precision: of what Evident drew, how much is real. Recall: of what's really there, how much it drew.
- Hallucination: the share of drawn items that match nothing in the labels. Routes the engine itself marks as dynamic (computed paths, low confidence) are drawn dashed and counted here, not as false positives.
- The benchmark repos so far are the author's own public projects. Labels are written from the source, and third-party repos are being added for the remaining adapters.
- Re-run it yourself: uv run evident-bench in engine/.