Rate limiting Fibre's read path without IP allowlists

[utku]

ADR-029 gave UploadShard a solid, stake-derived occupancy budget, and explicitly scoped the read path out: DownloadShard never grows the store, so it was left unthrottled. That gap is worth closing before Fibre carries real traffic, and I want to argue for a specific way to close it, because the obvious way (rate-limit by peer) doesn’t work here.

The problem with per-peer limits

DownloadShard is public to any client that completes the TLS handshake. No client cert, no mTLS, no request-level auth (see fibre_server.md, Transport Security). So the only thing a naive limiter has to key on is the connection: source IP, maybe ASN. That’s the standard move for a public read endpoint, and it’s the wrong one here, because it can’t tell apart the three kinds of traffic hitting this RPC:

1. Honest readers, fetching a shard they’re entitled to, for a blob that’s genuinely live.

2. Honest external observers, tools that re-derive shard assignment from chain state and deliberately query validators (including off-window, where NOT_FOUND is the correct answer) to check they’re actually serving what they signed for.

3. Abuse: scraping, DoS attempts, or a client that never checks whether a blob exists before asking every validator for it.

A peer-based limiter treats all three identically, because peer identity carries no information about which of these a request is. It also invites a fix that seems obvious and is actually harmful: allowlisting known monitor IPs. That lets a validator pass every check a monitor throws at it while quietly dropping real readers, and the selective serving becomes undetectable precisely because the only external check has a fast lane. Good external observability is a reason not to special-case the observer’s traffic.

What actually distinguishes the traffic

A legitimate request, reader or observer, has two properties that abuse usually doesn’t:

• It asks for a BlobID that corresponds to a real, currently-live publication, meaning an on-chain MsgPayForFibre settlement still inside its retention window.

• It asks the validator actually assigned that shard by the deterministic, stake-weighted assignment (Set.Assign). That’s public information, computable by any outside party from the commitment and the validator set, using the same deterministic function celestia-app runs.

Neither requires trusting or authenticating the client. Assignment is a pure function of on-chain data. I verified this independently, off-chain, bit-for-bit against the reference implementation across ~890 scenarios. Any honest client can do the same computation before opening a connection, and should, if it wants the shard on the first try.

The server already has this for free

This isn’t a new mechanism, it’s already in the code path. DownloadShard (fibre/server_download.go) decodes the BlobID, then calls store.Get(ctx, commitment). That lookup already answers both questions:

• Existence: no separate on-chain query needed. The store is the durable record of what got settled and admitted.

• Assignment: guaranteed by construction rather than by an extra check. UploadShard runs verifyAssignment before ever calling store.Put (fibre/server_upload.go). If a shard is in the store, this validator was the assignee at upload time. A hit on Get is a validated assignment.

The cost asymmetry is large too. Store.Get is a pebble prefix scan over small metadata keys; bulk shard bytes are deliberately kept out of pebble. A miss returns ErrStoreNotFound without touching a shard file. A hit reads the actual shard off disk, up to MaxShardSize (129.83 MiB per ADR-029’s sizing example), and serializes it back over gRPC. So “is this real” is nearly free, and “serve it” is expensive by orders of magnitude. That’s the shape a rate limiter wants: a cheap admission check gating an expensive payload.

What this suggests

Rate limit on the classification of the request, not on who’s asking.

A commitment genuinely in the store, within its retention window, is the honest-reader path, and functionally the same request as an observer’s. Leave it alone, or apply the loosest limit.

A commitment that was never real, or asking a validator never assigned it, is what abusive scanning looks like: hammering every validator hoping one sticks, without doing the free assignment computation first. Limit this hard per source. A client that skipped the free check is either broken or hostile.

NOT_FOUND after a publication’s must_serve_until has passed is also expected, and shouldn’t feed any abusive-peer score. It’s the shape of a well-behaved observer’s tail probe, checking that pruning happened on schedule.

For scale: in my own measurements a single probe (DNS, TCP, TLS and download, judged separately) runs p50 14ms / p95 24ms, with 4-5 points scheduled per assigned validator per publication. That’s a very different traffic pattern from a scraper hammering commitments that don’t exist. The tolerance for a missing shard also has to account for real pruning lag. I measured prune landing at pruneAt + ~1m45s rather than exactly at pruneAt, from the 60s prune loop plus the minute-granularity prune key. Without that, honest late prunes get flagged as something worse.

None of this needs new on-chain data, a new RPC, or client identity. It’s the same store.Get that already runs, read as a classifier instead of a binary success or failure.

Open questions

• Does this belong directly in fibre/server_download.go, cheap enough to always run, or as an operator-tunable wrapper?

• Should the rate itself derive from stake like ADR-029’s upload budget, or is a flat per-source limit fine for reads, given there’s no disk-occupancy risk, only CPU and bandwidth?

• Assignment constants (MinRowsPerValidator, LivenessThreshold, row counts) aren’t among x/fibre’s five governance params, they’re compiled into the validator binary. A classifier has to pull them from the binary version, so a silent cross-version mismatch is possible. That pinning needs deliberate design rather than a fixed table assumed to hold forever.

I can run this classifier logic against a multi-node devnet, with real publications, real assignment and real prune timing, plus a synthetic mix of valid, stale and wrong-validator requests, and come back with actual hit/miss and cost numbers for the three buckets above. Happy to do that pass if it would help move this past a proposal.