Fibre from the validator side (v10 lab) — Operator runbook & readiness checklist

Fibre from the validator side (v10 lab)

Network status

Mocha and Mainnet are app v9. x/valaddr / x/fibre are not in state.

celestia-appd tx valaddr set-host rejects until app version 10.

Operator note: Encoding is in CIP-51 / fibre/cmd/README.md. Reply with file + line if this is wrong.

Operator

  • Moniker:
  • valoper:
  • Region / DC:
  • Hosting: [provider]; if FDP: not Hetzner / OVH
  • Lab: [celestia-appd version] / [fibre version]

What has to run

Process Role Default
celestia-appd consensus + settlement app gRPC 127.0.0.1:9090
fibre shards, prune, client TLS 0.0.0.0:7980
privval gRPC endorse TLS + sign promises 127.0.0.1:26669

Clients dial only x/valaddr. No set-host = no traffic.

  1. set-host before v10 fails.
  2. --app-grpc-address and --signer-grpc-address are plaintext. Loopback or closed link only. Do not publish 9090 / 26669.
  3. Unbonded / not in the active validator set: storage budget 0. Process can be up and still receive nothing.

Client ↔ Fibre is TLS 1.3.

An Ed25519 TLS keypair is created on every Fibre start. The self-signed cert is endorsed by the consensus key and reminted on restart.

Registry accepts IP or DNS because identity is the consensus key.

--home is the store. Under it:

  • shards/
  • staging/
  • Pebble metadata

Do not delete or recreate this directory on restart or binary upgrade.

Prune

Every minute:

pruneAt = max(ExpiresAt, created + ShardRetention)

Default ShardRetention = 4h unless params change.

CIP-51 defines fibre.FullStakeStorageBudget = 2 TiB by default; CIP-52 draft lists that default at v10 initialization.

A given validator’s budget is stake-derived and governance-changeable.

Headroom on disk is still needed for staging + Pebble.

Do not treat 2 TiB as a machine spec.

Remote signer must implement SignRawBytes.

Docs target:

median cometbft_privval_signing_latency_* ≤ 10ms

Lab — v10 local only

Do not point these at Mocha.

After make build-fibre-server, use ./build/fibre so $PATH cannot pick an older install.

If the release binary is already at /usr/local/bin/fibre, that path is fine.

make build-fibre-server

./build/fibre version

./build/fibre start \
  --home /var/lib/celestia-fibre \
  --app-grpc-address 127.0.0.1:9090 \
  --server-listen-address 0.0.0.0:7980 \
  --signer-grpc-address 127.0.0.1:26669

Helpers:

scripts/single-node-fibre.sh
scripts/submit-fibre-blobs.sh
tools/fibre-txsim

Register the Fibre host

celestia-appd comet show-address

celestia-appd tx valaddr set-host 127.0.0.1:7980 \
  --from <operator-key> \
  --chain-id <local>

celestia-appd query valaddr provider <celestiavalcons...>

Host format:

host:port

Requirements:

  • port 1–65535
  • max 100 chars
  • no scheme
  • no path

Pass

  • query returns the host
  • submit exits 0
  • reconstruct matches

Fail

Typical causes:

  • still v9
  • privval gRPC off
  • bind 127.0.0.1 while the client is elsewhere
  • skipped set-host
  • signer changed and Fibre not restarted

Please include these when reporting a lab result:

[celestia-appd version]
[./build/fibre version]
[query valaddr provider]
[local set-host tx hash]

systemd

Create and chown the home first, or use StateDirectory.

With StateDirectory, systemd owns:

/var/lib/celestia-fibre

Example unit:

[Unit]
Description=Celestia Fibre server
Wants=network-online.target
After=network-online.target celestia-appd.service
Requires=celestia-appd.service

[Service]
User=celestia
StateDirectory=celestia-fibre
ExecStart=/usr/local/bin/fibre start \
  --home /var/lib/celestia-fibre \
  --app-grpc-address 127.0.0.1:9090 \
  --server-listen-address 0.0.0.0:7980 \
  --signer-grpc-address 127.0.0.1:26669
Restart=always
RestartSec=3
LimitNOFILE=65535

[Install]
WantedBy=multi-user.target

Prepare 7980/tcp on the firewall.

Do not systemctl start fibre against Mocha/Mainnet while they are v9.

Keep /var/lib/celestia-fibre across upgrades.

Order on a v10 chain

appd healthy
    ↓
app / privval gRPC reachable
    ↓
start Fibre process + listen
    ↓
set-host

Registering a dead host makes clients time out.


Day the network is v10

Before height

Have the following ready:

  • backup
  • matching appd + Fibre
  • privval gRPC configured
  • persistent Fibre home
  • systemd unit installed
  • firewall prepared for 7980
  • public host:port

Do not start Fibre against the v9 network.

After height

Expected order:

app v10 and bonded
    ↓
gRPC healthy
    ↓
Fibre listening
    ↓
set-host
    ↓
query valaddr provider
    ↓
record UTC + tx hash

GC

The record is dropped if the validator:

  • leaves staking, or
  • is jailed and unbonded for more than 7 days

Run set-host again only after you are back and the record is gone.

Ops target

We will run Fibre + set-host within 24h of Mocha v10 on:

[host / region]

That is our ops target, not an extra FDP clause.

FDP text is:

  • run Fibre once it is live
  • upgrade in 24h
  • no Hetzner/OVH

Five checks

# Check Mocha/Mainnet today Local v10 lab
1 Fibre process install / version-check only; do not start yes
2 :7980 listen n/a until network v10 yes
3 valaddr host n/a — no module yes
4 shard I/O n/a yes, generate it
5 filesystem of --home < 80% operator alert, not a CIP rule same

Public label until network v10:

Fibre: waiting.

Watch:

  • journal
  • prune
  • store
  • sign

There is no first-class Fibre gauge for disk %.

Use node exporter for filesystem monitoring.


Breakage / troubleshooting

Seen Do
0 traffic, process up check registry, firewall, bind 0.0.0.0, bonded status
set-host rejected still v9 / bad host:port / wrong key
query empty wait for commit; run comet show-address
creating signer check privval gRPC
storage budget is 0 validator not bonded / not in active set
TLS errors after key change restart Fibre to generate a new cert
disk full check prune loop; do not wipe $FIBRE_HOME
jailed unjail; query registry; run set-host only if GC’d

Server config exposes:

  • MaxMessageSize
  • UploadVerifyWorkers

There is no per-peer rate limiter in the server spec.

A normal TLS-terminating proxy will break Fibre validator-identity verification.

Use TCP/L4 passthrough unless the intermediary implements Fibre-endorsed TLS identity.


Do not

  • Call the lab Mocha or Mainnet.
  • Run set-host on v9.
  • Start Fibre against Mocha/Mainnet while they are v9.
  • Publish 9090 / 26669.
  • Wipe $FIBRE_HOME to “fix” a restart.
  • Put the FDP Fibre host on Hetzner/OVH.
  • Publish 1 Tb/s or 2 TiB as your SLO.

Questions

  1. If Fibre is not on the appd box, how are you fencing plaintext 9090 and 26669?

  2. Which remote KMS actually does SignRawBytes, and does signing stay ≤ 10ms across restart / cert remint?

  3. How are you watching store occupancy + prune health? Metrics cover prune/store latency, not filesystem %.

  4. Is 7980 raw to the internet, or L4 passthrough / limiter?

  5. Lab set-host: IP or DNS, and what is the procedure when the box moves so the registry is not stuck on a dead endpoint?

If Fibre is not on the appd box, how are you fencing plaintext 9090 and 26669?

We will implement TLS in the upcoming month for the privval endpoint.

For securing the endpoints if you’re not running Fibre in the same server as the validator, you could use a private network, wireguard tunnel or implement a firewall

Which remote KMS actually does SignRawBytes, and does signing stay ≤ 10ms across restart / cert remint?

Both TmKMS and Horcrux support the SignRawBytes method. It’s fine if latency is higher for some time for operational actions.

How are you watching store occupancy + prune health? Metrics cover prune/store latency, not filesystem %.

That’s for the operator to figure out. Feel free to open PRs to add metrics that cover this.

Is 7980 raw to the internet, or L4 passthrough / limiter?

It should be exposed. We’re currently working on rate limiters for this.

set-host: IP or DNS, and what is the procedure when the box moves so the registry is not stuck on a dead endpoint?

You could use any. DNS would make it easier for you to switch backends without needing to update the x/valaddr registry. However, if you need to change that, you can submit a transaction to update the registry and that’s it.