Replacing the network under a running production fleet

The situation

TODO(anshuman): rewrite this from your actual engagement. The structure below is the one to keep — it's what makes a case study read as senior rather than as a list of tasks.

A production Kubernetes fleet running a legacy CNI. Two problems had converged: the network policy model could not express the segmentation an upcoming compliance requirement demanded, and during incidents nobody could see east-west traffic — every investigation started by guessing which service was calling which.

The constraint

[[What made this hard? Some candidates — pick the true ones:]]

• Changing a CNI on a running cluster is not a supported in-place operation
• [[N]] production clusters serving customer traffic with no maintenance window
• The kernel version on existing nodes predated the eBPF features required
• A rollback after cutover would be far more expensive than a rollback before it

What I did

[[The approach, and — importantly — the options you rejected and why.]]

I ran it as a staged, node-pool-by-node-pool migration rather than a cluster replacement. New node pools on an updated kernel, Cilium installed in a mode that could coexist with the incumbent CNI during transition, workloads drained across a pool at a time, with a defined go/no-go check at each stage and a documented rollback that stayed cheap until the final pool.

Options considered and rejected:

• Blue/green cluster replacement — cleaner in theory, but required duplicating stateful infrastructure and a DNS cutover with a much larger blast radius.
• Stay on the incumbent CNI and add a separate observability layer — lower risk, but two systems to operate and no kernel-level policy.

The decision and its trade-offs were recorded as an architecture decision record so the reasoning survived the project.

The outcome

• Completed with no customer-visible downtime
• Kernel-level L3/L4 policy replaced a userspace proxy layer
• Hubble flow visibility surfaced [[N]] service-to-service dependencies the team did not know existed — two of which were removed as a result
• [[Any latency or resource change you measured]]

What I'd do differently

[[Optional but valuable. What surprised you? What would you sequence differently? This section signals more experience than the rest of the page combined.]]