Highly available blockchain validators on a multi-cloud infrastructure

Multi-cloud re-architecture (GCP, OVH, Hetzner) for a blockchain validator operator: −50% costs, zero slashing and zero jail across Cosmos, Ethereum and L2s.

EXP-08Web3

Client: Node Guardians
Role: Web3 DevOps Engineer
Period: Feb 2023 → Sep 2024
Status: Completed
Stack: KubernetesTerraformAnsibleGCPOVHHetznerGitHub ActionsPythonPrometheusGrafanaLoki

−50%

infrastructure costs

slashing / jail events

Context

Node Guardians operates validators for many blockchains. In this business, any downtime is paid for in on-chain penalties (slashing, jail): the infrastructure has to be highly available, secure and cost-effective all at once.

I designed the new multi-cloud architecture, migrated the entire legacy infrastructure, and built the team's internal Python tooling.

Multi-cloud infrastructure

Architecture spread across GCP, OVH and Hetzner for resilience and cost.
Full migration off the legacy infrastructure without interrupting the validators.
Production Kubernetes clusters: deployment, hardening, operations.
Terraform and Ansible IaC with custom modules; GitHub Actions CI/CD.
Proactive observability: Prometheus, Grafana, Loki, Sentry.

Web3 validators

Cosmos ecosystem: Cosmos Hub, Osmosis, Axelar, Kyve, Celestia, dYdX.
Ethereum: highly available validators with client diversity.
Layer 2: Kroma and Arbitrum sequencers; other chains: Avalanche, SUI, Aligned Layer.

Internal Python tooling

Internal CLIs and APIs: Click, Pydantic, SQLAlchemy, FastAPI.
PEP8-compliant code with a testable architecture.

Results

Infrastructure costs cut in half despite a growing workload.
Zero slashing and zero jail for the entire engagement, thanks to a resilient architecture and proactive monitoring.

Client website