Method · enforced by bench/rbench

Methodology

The rules every run follows, why each exists, and where the verifier enforces it. They are fixed before any data is collected.

Four quantities, never reported as one

Quantity	Measured as	Who can check it
Inclusion	Δslot = landed slot − anchor slot, from the chain	Anyone, from chain
Fill quality	Realized price vs the curve price at the anchor slot, from transaction balances	Anyone, from chain
Internal processing	Engine monotonic-clock spans on one host, in ns	Operator's own clock
Detection lead	First-seen vs the event's slot; needs shred-slot logging	Operator's own logs

A slot is about 400 ms and the order inside it belongs to the leader, so landing in the same slot proves no millisecond figure. Δslot is never converted to milliseconds. Millisecond spans appear only as engine-reported, labelled “not chain-verifiable”.

Rules shared by every experiment

Topic	Rule	Why
Unit and arms	One arm per intent: arm = arms[sha256(run_id:seq) mod k], or per launch by mint. The verifier recomputes every assignment.	Two arms on one token in one slot contend for the same bonding-curve account. And the operator is the party a sceptic doesn't trust.
Timing	Probes exist only at hash-scheduled slots; trigger sets are recomputed from chain data. Missing, extra or duplicate intents fail the run.	Otherwise the engine could send only when conditions favour it.
Anchor slot	Every Δslot is measured from an on-chain anchor named in the protocol: scheduled probe slot, trigger tx or threshold cross.	“Same slot” needs a reference.
Denominator	Every intent created, including refused, expired, engine-error and not-landed.	Survivorship.
Tip	A fixed tip per arm, at or above each provider's minimum. A tip sweep is a separate protocol.	Sub-minimum tips are silently dropped and would look like slow landing.
Strata	Congestion tercile, leader-window position (slot mod 4, computed by the verifier), region, launchpad. Every stratum is reported.	Results that hold only when quiet aren't results.
Survivorship	Tokens that rug or migrate mid-run stay in the data with their outcome class.	Dropping them keeps only the easy cases.
Commitment	finalized by default.	A confirmed transaction can still roll back.

Statistics

Implemented in rbench/stats.py and tested against published reference values.

Question	Method	Why this one
A rate (landed within the window)	Wilson score interval	Behaves near 0 and 1, where a Wald interval breaks
Difference between two randomized arms	Newcombe hybrid score	The right interval for unpaired proportions. Overlapping individual intervals is not a test.
Correlated intents	Cluster bootstrap over blocks of cluster_slots; decisions use the wider interval	Intents close in time share leaders and congestion
A quantile of Δslot	Distribution-free order-statistic interval	Returns nothing when n is too small: p99 at 95% needs n ≥ 368, and at n = 1,000 it spans ranks 982–996
Shift in Δslot between arms	Hodges–Lehmann with a seeded percentile bootstrap	Robust to the heavy right tail; always shown beside the landing difference
Planning n	Two-proportion power × design effect 1 + (m−1)·ICC	Inputs are written into the protocol; ICC comes from a pilot

When pairing is valid

Only for copies of one signed intent racing to one leader, under durable nonce: exactly one copy can land, so route win share is a within-intent comparison. Everything else uses randomized single-arm assignment and unpaired methods.

How uncertainty is shown

Every number carries k/n and an interval, or the word indicative with the reason.

Conditional metrics travel together. A Δslot shift is conditional on landing, so it always sits next to the landing-rate difference: a faster arm that lands less often hasn't won.

Each run is analysed once. If interim looks are needed, the protocol names the stopping rule before the run. Strata are descriptive unless the protocol says otherwise in advance (Holm correction).

There are no gauges, no “× faster” hero numbers, and no chart without its n.

How cherry-picking is made impossible

Requirement	Mechanism	Enforced by
Pre-registered definitions	Finalize → hash → post the hash before intent 1	Verifier refuses on a hash mismatch
No arm shopping	Arm recomputed from sha256(run_id:seq)	tests: test_arm_shopping_is_detected
No hidden intents	Gap-free seq; every scheduled slot present	tests: test_hidden_intent_is_detected
No chosen timing	Hash-scheduled probes; anchors re-fetched from chain	tests: test_unscheduled_anchor_is_detected
Claims match evidence	claimed_metrics.json must equal the recomputation	verify exit code 2
No quiet re-runs	A run registered internal can never be published public	rbench.publish
Sensitive data protected	Per-tier redaction; signatures, slots and ids never redacted	rbench.publish

Three tiers of evidence

Tier	Who sees it	Gate to move up
internal	Rout team: dev runs, devnet, failed experiments	—
partner confidential	A partner under NDA; their own launch data stays here unless they approve otherwise	Registered at this tier or wider; partner consents
public	This site. Mints, sizes, quotes, regions and error strings are removed; signatures, slots and ids stay.	Registered public before the run; release approved

What these benchmarks don't show

E0 probes are non-trading self-transfers. They take no write lock on a hot bonding curve, so they show how good the submission path is, not how buys land under contention. Every E0 chart says so.

Guard and exit runs use positions Rout is allowed to trade. Your traffic may differ, which is why a partner can run the verifier on their own launches.

These are not data-feed benchmarks. A feed benchmark measures when information arrives; these measure whether a transaction lands, when and at what cost. A fast feed says nothing about landing, and a tip or route choice says nothing about feed speed.

Nobody else's clock is used. Feed and sender figures elsewhere are often measured from the vendor's own timestamp, which leaves the vendor's own delay out. Every delay here starts at an on-chain slot.

Rout operates these runs. The verifier exists so you don't have to trust that.