// accuracy
Our own numbers, not a scoreboard
No Sybil-detection vendor publishes their own false-positive rate. Here are ours — self-reported, methodology attached, fully reproducible. This page is deliberately not a “SybilShield vs. Trusta vs. Nansen” leaderboard — see why below. For a feature-by-feature comparison, see /compare.
Preset calibration retro — 600 real wallets
Before shipping the airdrop preset, we ran it against 600 addresses with real ground truth — and found it flagged 66% of confirmed genuine governance voters. Full writeup: /blog/preset-calibration.
| sample (n=200 each) | before calibration | after calibration |
|---|---|---|
| LayerZero T1 (confessed sybils) | 200/200 DROP — 100% recall | 200/200 DROP — 100% recall |
| Arbitrum T4 (sybil list) | 200/200 DROP — 100% recall | 200/200 DROP — 100% recall |
| Governance G2 (confirmed genuine voters) | 132/200 DROP — 66% false-positive | 0/200 DROP — 0% false-positive |
The “after” column's 0% is specifically the hard-DROP false-positive rate. 60 of the 200 genuine voters (30%) still landed in REVIEW, not a clean pass — real cluster co-residents (shared funder or behavior), some plausibly legitimate (multisig signers, delegate aggregators), routed to manual review rather than auto-cleared or auto-dropped. That's what REVIEW is for.
ML holdout metrics — model v0.5.0-gov-expanded
Held out 30 sybil (T1+T2 hand-verified) + 30 genuine (G2) addresses, never seen at training. Full writeup: /blog/v05-real-corpus.
n_train=900 · n_holdout_sybil=30 · n_holdout_genuine=30. Adversarial recall was 0.000 on the previous model version — this is a real improvement, not a static number.
What these numbers do NOT mean
- · Not wild-traffic-calibrated. Every number above is honest-holdout, on a small, curated, hand-labelled set — not a claim about your address list. The 60-address ML holdout split is trivially separable by construction (confessed sybils vs. governance voters look very different even before feature engineering); a much simpler classifier would likely score similarly on this exact split.
- · The genuine pool is still small (~1,700 addresses). A brand-new wallet with no governance history looks statistically different from our current “this is fine” examples — false-positive rate on small/new accounts is a genuinely open question until wild-traffic feedback exists.
- · These numbers will move. As the genuine corpus grows (Gitcoin Passport G1 integration is on the roadmap — ~1,700 → ~50,000) and real customer feedback accumulates via the thumbs-up/down on every verdict, expect these figures to be re-published, not quietly left stale.
Why no competitor numbers here
We don't have access to any other vendor's ground truth, methodology, or real false-positive rate — publishing a number we can't actually verify, attached to another project's name, isn't honest benchmarking, it's a claim we can't back up. If you run a filter today, the retro above is written so you can reproduce it against your own: pick a sample of confirmed genuine governance voters from your chain's biggest DAOs (all public data) and check what fraction your filter flags. If it's above a few percent, you likely have the same root cause we found — shared-funder clusters from ordinary CEX withdrawals being treated as sybil signal.
Every method, threshold, and the model artifact's hashes are open-source and reproducible — see /methodology and the public repo. Found a wallet we called wrong? File an appeal — 48h SLA, full audit trail.