Moving Fibre off the validator host: private VLAN + mTLS (mocha-5)

Moving Fibre off the validator host: private VLAN + mTLS (mocha-5)

On 8 October we moved our mocha-5 Fibre off the validator host onto a separate machine in the same datacentre, linked by a private VLAN with mutual TLS on the signing channel. A short write-up of how we did it, which may be useful to validators planning to split Fibre from the validator node.

As an alternative we considered a WireGuard tunnel, but our provider offers a private network between servers, so for us that was the optimal option.

Choosing the server

We sized the machine from Fibre’s requirements, with headroom for growing load:

  • CPU. The Foundation’s requirements call for a CPU with GFNI and SHA-NI. For Fibre the recommendation is 8-16 high-clock cores, or better yet a CPU that passes tools/cpu_requirements. We checked for the instructions on the actual server before ordering: vendor pages do not always list them.
  • Memory. Fibre holds gRPC buffers: max_connections Ă— max_concurrent_streams Ă— 132 MiB, about 27 GiB in the worst case with defaults. It does not depend on stake, so 32 GB.
  • Storage and traffic. These depend on stake and on the network’s target throughput. At the 148-row floor and current parameters that is 79.5 GB and 0.053 Gbps of ingress. The Foundation has a calculator for future parameters: at the possible 2.2 GB/s target for mainnet the same floor needs about 1.2 TB and about 255 TB of traffic per month, so leave headroom on disk and traffic quota.

Private network or WireGuard

The app gRPC channel between Fibre and the node is unencrypted, so it must stay on the same host or a trusted private network. Our provider offers a free private VLAN between dedicated servers on the same account, with no speed limit. It runs over the same network ports as the public network, so no extra hardware is needed.

The signing channel is protected by mTLS, and the app gRPC and signer ports listen only on the VLAN address. If your provider has no private network between servers, charges for it or has other limitations, WireGuard is one option. A tunnel will probably matter even more once a validator can run several Fibre servers in different places.

As soon as the signer address is not local (loopback), the node requires mTLS and will not start without it: this is what protects access to signing with the consensus key. We made our own CA, a server certificate for the node with the validator’s VLAN address in the SAN, and a client certificate for Fibre with its VLAN address in the SAN.

Cutover

The provider sets up the VLAN on its own equipment and warned us in advance about a brief interface flap. We did not ask for a dedicated window, because we had worked out the margin. In our case each of the server’s two links went down for 5-7 seconds, one after the other, and the validator missed only 10 blocks: a blip this short is entirely safe and causes no meaningful downtime.

Then the order is:

  1. Stop the old Fibre on the validator.
  2. Move the signer and app gRPC addresses to the VLAN address with mTLS, restart the node, confirm it signs.
  3. Start the pre-prepared Fibre on the new server and check its log for signer ready and transport=mtls.
  4. Important: once Fibre is listening on port 7980, register the new server’s public (not VLAN) address with celestia-appd tx valaddr set-host <new server's public address>:7980 from the validator account, otherwise clients keep going to the old address. The registry accepts an IP or a DNS name: with a DNS name the next move needs no new transaction, just a DNS record update.

After setup, check with a reboot that the private interface comes up automatically.

What changed

1. Fibre on the validator → a separate server. The signing channel moved from local loopback to a private VLAN with mTLS. Latency to clients did not change: the TLS handshake from our monitors was 0.587 s before the move and 0.589 s after. Today there is no visible difference in operation: both machines are in the same datacentre, and at minimal load Fibre in our case uses about 0.3% CPU and 185 MB of memory. The benefits of the split will show under load.

What the split removes: when Fibre runs on the validator host, under load the two share resources. That means Fibre’s buffers of up to 27 GiB of memory, CPU for verifying uploads, disk and network bandwidth, and Fibre’s public port is open on the validator’s machine. At today’s load this is not critical, but as Fibre traffic grows these are the resources that can start competing with consensus.

2. Warsaw → Singapore, Melbicom. Before this, for decentralisation (both geographic and across providers) and in line with the recommendation to move outside the EU and North America, we moved our mocha-5 validator from Warsaw to Singapore, to Melbicom.

We already had a positive experience with Melbicom. Support answers any question in detail even before you order: at our request their sales team checked the CPU flags on a specific server. The validator server has datacentre-class Samsung PM9A3 NVMe drives and two 10 Gbit ports in aggregation (20 Gbps with link redundancy), and the private VLAN between servers is free.

The price of the move is distance from most of the network in Europe. By our data, latency to Fibre hosts in Poland has a median of 70 ms, to ours in Singapore 420 ms; from our monitors in Europe, Singapore is about 190-200 ms away. It is too early to say how the distance will affect Fibre: there are few uploads on mocha so far. We are tracking it.

What’s next

We expect load on Fibre to grow. We will track it with metrics and share the results once the numbers start to mean something. We are also waiting for the ability to scale Fibre across several servers for one validator: it is not supported yet, but this capability is already described in issue #7690. A separate Fibre server is our first step in that direction.

1 Like