Replacing the network under a running production fleet

The situation

A production Kubernetes fleet — four clusters, nine node pools between them — running a legacy CNI. Two problems had converged.

The first was regulatory. An upcoming compliance requirement demanded segmentation the incumbent policy model could not express: the client needed to prove that a workload processing one category of data could not reach a service handling another, and the available primitives could only say "these labels may talk to those labels" at a granularity that didn't survive the question.

The second was operational, and older. During incidents nobody could see east-west traffic. Every investigation started the same way — someone guessing which service was calling which, then proving or disproving it by reading code. The team had learned to work around this so completely that they no longer described it as a problem.

The second one is why I was comfortable recommending the change. A migration justified only by a compliance date gets deprioritised the moment the date moves. One that also fixes something the team feels every week does not.

The constraint

Changing a CNI on a running cluster is not a supported in-place operation. It is not a configuration change; it is a replacement of the component that gives every pod its address.

That was compounded by three things:

• No maintenance window. The platform serves customer traffic continuously and the contractual availability target left no room to schedule one.
• Kernel versions predated the required eBPF features on most existing nodes, so this was a node replacement exercise as well as a CNI one.
• Rollback economics were asymmetric. Backing out before cutover was cheap. Backing out after would have meant a second unplanned network migration under time pressure — the worst possible conditions for the work.

That asymmetry shaped the entire plan. The goal was not to avoid failure; it was to keep failure cheap for as long as possible.

What I did

A staged, node-pool-by-node-pool migration rather than a cluster replacement.

New node pools were provisioned on an updated kernel with Cilium installed in a mode that could coexist with the incumbent CNI during transition. Workloads were drained across one pool at a time, with a defined go/no-go check at each stage — policy enforcement verified, flow logs sane, error rates and p99 latency inside the pre-agreed band — and a documented rollback that stayed cheap until the final pool was cut.

Nine pools, nine decision points. Two of them were paused and resumed the following day after the check showed something worth understanding first. Neither was an incident, because pausing was one of the planned outcomes rather than a deviation from the plan.

Options considered and rejected:

• Blue/green cluster replacement. Cleaner in theory. It required duplicating stateful infrastructure and finishing with a DNS cutover whose blast radius was the entire platform at a single moment. Trading nine small risks for one large one is usually a bad trade, and it is always a bad trade when the large one is hard to reverse.
• Stay on the incumbent CNI and add a separate observability layer. Lower risk and genuinely tempting — it would have solved the visibility problem, which was the one the team felt. But it left the compliance requirement unaddressed and added a second network-adjacent system to operate. It buys a year and costs the same migration later, with more running on it.

The decision and its trade-offs were recorded as an architecture decision record, so the reasoning survived the project. Six months later, when someone asks why the platform isn't on the CNI their previous employer used, the answer is a document rather than an argument.

The outcome

• Completed with no customer-visible downtime across all nine pools
• Kernel-level L3/L4 policy replaced a userspace proxy layer, and the segmentation the compliance requirement asked for became expressible — and demonstrable to an assessor
• Hubble flow visibility surfaced 23 service-to-service dependencies the team did not know existed. Two were removed as dead paths. One was a test harness that had been calling production since 2023.
• Per-node CPU overhead from the networking layer dropped noticeably once the proxy layer was gone, though the honest framing is that this was a welcome side effect rather than a goal — nobody had been complaining about it

The dependency discovery was the outcome the client talked about afterwards, and it was the one nobody had put in the business case. That is common enough to be worth planning for: the visibility you gain during a migration is often worth more than the capability you migrated for.

What I'd do differently

I'd run Hubble in observation mode before committing to the migration, not during it. We discovered the 23 unknown dependencies while migrating, which meant discovering them under time pressure with a rollback clock running. The same information a fortnight earlier would have been free, and it would have made the policy design better rather than making the migration more anxious. Cilium can be installed in a flow-visibility-only posture without taking over networking; there was no reason not to do that first except that I was thinking about the migration as one project rather than two.

I'd have pushed harder on the kernel upgrade as its own piece of work. Bundling it into the CNI migration meant every go/no-go check was evaluating two changes at once. When a pool needed a pause, the first hour went on working out which of the two changes was responsible. Separating them would have added elapsed time and removed genuine ambiguity — a trade I'd take again.