Almost everything I build on the side runs on my K3s cluster at Hetzner: this homepage, the advent calendar, URL Unicorn, runcard and a few more. Until yesterday it also ran Kubernetes 1.27, which has been out of support since 2024. As long as everything worked, I had no reason to touch it.
Then Argo CD 3.6 arrived. It is tested on Kubernetes 1.34 to 1.37, and the advent calendar starts on 1 December. So the upgrade had to happen now, while an outage would only annoy me and not the people playing the calendar.
Nine hops, not one jump
Kubernetes does not let the API server skip a minor version, not even on a single node, and a kubelet must never be newer than its API server. From 1.27 to 1.36 that means nine upgrades, each to the latest patch of the next minor: the server first, then the agent.
Why stop at 1.36? 1.34 is the first version in Argo CD's range, but it went out of support at the end of October. 1.36 is supported until June 2027, and cert-manager and external-secrets both cover it. 1.37 would have added another etcd minor, and cert-manager does not list it yet.
| Hop | Kubernetes | What changes |
|---|---|---|
| 1 to 3 | 1.28 to 1.30 | Traefik 2.11, Flannel 0.27 |
| 4 | 1.31 | containerd 1.7 to 2.1 |
| 5 | 1.32 | Traefik v2 to v3, the old Traefik CRDs are deleted |
| 6 | 1.33 | etcd 3.5 to 3.6, no way back but a snapshot |
| 7 | 1.34 | first version in Argo CD 3.6's range |
| 8 and 9 | 1.35 and 1.36 | containerd 2.3, the target |
Reading before upgrading
The day before, Claude Code and I went through the cluster read-only and wrote a plan: for every hop, which Traefik, etcd, containerd, Flannel and CoreDNS versions k3s bundles, what Kubernetes removes, and what could break on this cluster in particular. Two findings made that day worth it.
👻 A node that had been dead for six weeks. kube-node-2 had stopped reporting on 1 October, about two and a half hours after it joined, and I had not noticed. It is gone now, and the cluster runs on two nodes: one server with embedded etcd and one agent.
🚦 A redirect that would have taken down almost every site. The hop to 1.32 swaps Traefik v2 for v3, and the new CRD chart deletes the old traefik.containo.us API group. My http-to-https redirect Middleware existed only in that group. I had applied it by hand in December 2024 and never committed it. Every Ingress except the one of Argo CD references it, and Traefik does not create a router whose middleware is missing. Without preparation, every site except Argo CD would have answered 404 after hop 5.
The fix was the same Middleware in the new traefik.io group, applied a day early:
apiVersion: traefik.io/v1alpha1
kind: Middleware
metadata:
name: redirect-https
namespace: default
spec:
redirectScheme:
scheme: https
permanent: true
Traefik 2 already prefers the traefik.io object when both exist, and Traefik 3 reads only that one, so the name resolved at every moment of the hop. The manifest is in Git now.
A script with gates
Nine hops by hand means typing the same twenty commands nine times and getting one of them wrong at some point. So the plan became a script, hop.sh, that runs one hop in phases: checks before, an etcd snapshot copied off the node, the server install, the agent install and checks after. Every phase ends in gates, and the script stops at the first gate that fails. After a fix, START=<phase> resumes where it stopped, and START=check runs every gate read-only against the current state.
The gates check the node versions, the API server, the k3s chart jobs, the expected Traefik image, every rollout and pod, all Argo CD Applications, Certificates and ExternalSecrets, DNS and the API from the other node, and the http and https status of all 49 hosts. The last one is compared against a baseline taken before the first hop. The rule is not "200 everywhere", because some APIs have no root route and answer 404. The rule is "every host answers exactly as before".
Two more details saved me. The k3s install script rewrites the systemd unit from its arguments, so every server flag must be passed again, and the script diffs the unit afterwards to prove nothing was lost. And the installs run on the node under systemd-run --wait, so a dropped SSH connection cannot kill an install halfway.
The run
The plan suggested four sessions over a few weeks: the easy hops first, then Traefik v3 alone with a few days to watch certificate renewals, then etcd. When it came to the timing question, my answer was "Everything now!". 😅
Before the two risky hops, Traefik v3 and the etcd jump, I took a Hetzner snapshot of the server in the console myself. Each hop took about five minutes including all checks. Between the hops cert-manager went from 1.16 to 1.21 in five steps and external-secrets from 1.3 to 2.13, each as soon as the cluster version was in its supported range. Meanwhile a second agent read the urgent upgrade notes of Kubernetes and the release notes of k3s, cert-manager and external-secrets for every target in the background. It found no blocker.
The gates stopped the run twice:
- ⏰ Hop 1: a CronJob started in the very second the agent restarted. Its first pod timed out on the database, the retry succeeded. Not a problem in the cluster but a gate that was too strict: failed pods of a Job that succeeded are ignored now.
- 🔀 Hop 5: seven hosts changed from 404 to 503. They belong to the 2024 and 2025 advent calendars, whose backends are scaled to zero. Traefik v2 skipped a Service without endpoints entirely, Traefik v3 creates the router and answers "no available server". Expected, so the new answers became the baseline.
After the switch to Traefik v3 I also issued a throwaway Let's Encrypt certificate to prove that the HTTP-01 challenge still works. It was issued in about 25 seconds.
Fifty-two minutes after the first hop both nodes ran k3s 1.36.6 with etcd 3.6, containerd 2.3 and Traefik 3.7. All 49 hosts answered as before, and all 23 Applications, 39 Certificates and 33 ExternalSecrets were in place. Pods keep running while k3s itself restarts, so the hops only cost short blips whenever the single Traefik pod was replaced.
Cleaning up
Nine upgrades leave a lot behind, and the agent node was at 73 % disk afterwards. Four steps brought it back:
- 🗑️ removing every image no container uses, 140 down to 32 on the agent
- ♻️ kubelet image garbage collection through a config drop-in, starting at 70 % disk and cleaning down to 50 %, plus a maximum age of 14 days for unused images
- 📓 journald capped at 1 GB, it had grown to 4 GB per node
- 📦 the old k3s data directories, because k3s extracts one per version and never deletes them
The last step needed care. Pods started before the upgrade still run their containerd shims from the 1.27 directory, some of them since April 2025. Deleting that directory would break kubectl exec and a clean shutdown for those containers, so only directories that no running process references were removed. In the end the server went from 43 % to 25 % disk and the agent from 73 % to 23 %.
What I learned
📜 Everything belongs in Git. The one object I had applied by hand two years ago was the one that would have broken nearly every site.
📏 Compare against a baseline, not against an ideal. "Same as before" catches real regressions. "200 everywhere" would have failed on the first hop and been ignored from then on.
🚧 Gates turn repetition into a procedure. Without the script I would have stopped after three hops and continued "next month", and the cluster would still be on 1.30 a year from now.
🧹 kubectl apply never deletes anything. cert-manager is installed from its static release manifest, and applying a newer one left a Role and two RoleBindings behind that upstream had renamed or dropped. I diffed the object lists and deleted them by hand. Helm does not have this problem.
The cluster is now on a version that is supported until June 2027. Next on the list is the system-upgrade-controller, which can keep it on the 1.36 patches, so the next upgrade is a non-event instead of a project. And the server has not rebooted for almost two years, but that is a story for another post. 🙈
K3s, Kubernetes, etcd, containerd, Traefik, cert-manager, external-secrets, Helm, Argo CD
Hetzner, Ubuntu, systemd, Bash, Claude Code
Kubernetes version skew policy, the k3s upgrade and rollback docs, the Traefik v2 to v3 migration guide