Upgrade control plane Kubernetes

Spaces upgrades the Kubernetes version of existing control planes to the latest version the Spaces release supports. Control planes upgrade one minor version at a time and must pass health checks before each next step.

Settings

Setting	Where	Default	Description
controlPlanes.kubernetesUpgrade.paused	Helm value	false	Pauses upgrades for every control plane in the Space.
controlPlanes.kubernetesUpgrade.maxParallel	Helm value	20	Maximum number of control planes that upgrade at the same time.
spaces.upbound.io/kubernetes-upgrade-paused: "true"	Label on a ControlPlane or its host namespace	Unset	Pauses upgrades for one control plane.
spaces.upbound.io/kubernetes-upgrade-resume	Annotation on a ControlPlane	Unset	Retries a halted upgrade when its value changes.

Pausing stops new minor version upgrades. A minor version upgrade that already started still finishes or halts. Patch updates still apply.

Spaces 1.19 removes the controlPlanes.k8sVersion Helm value. Remove it from your Helm values before you upgrade. To control when upgrades start, set controlPlanes.kubernetesUpgrade.paused=true in the same helm upgrade that moves the Space to 1.19.

See the Helm chart reference for all Spaces Helm values.

How upgrades proceed

Before each minor version upgrade, Spaces checks that the control plane is healthy. The checks cover the vCluster and etcd pods, the API server's /readyz endpoint, and the Crossplane deployment in the control plane. They don't check your workloads. After the upgrade rolls out, the checks must pass for three minutes in a row. An upgrade that doesn't get there within 30 minutes halts.

At most controlPlanes.kubernetesUpgrade.maxParallel control planes in a Space upgrade at the same time. The rest report UpgradeQueued and start oldest first, by creation time. A halted control plane counts toward maxParallel until you resume it and it finishes, or until you pause its upgrades. If maxParallel control planes halt, no other control plane in the Space starts upgrading until you act.

To try a release on one control plane first, set maxParallel to 1. To upgrade some control planes last, pause them until the others finish.

Pause upgrades for a control plane

kubectl label controlplane <name> -n <group> \
  spaces.upbound.io/kubernetes-upgrade-paused=true

To resume upgrades, remove the label:

kubectl label controlplane <name> -n <group> \
  spaces.upbound.io/kubernetes-upgrade-paused-

A pause set on the host namespace or the Space still applies after you remove this label. The KubernetesUpgrade condition message lists every pause in effect.

Check upgrade status

Each ControlPlane reports upgrade progress in its KubernetesUpgrade condition.

kubectl get controlplane <name> -n <group> \
  -o jsonpath='{.status.conditions[?(@.type=="KubernetesUpgrade")]}'

Reason	Meaning
UpToDate	The control plane runs the latest supported version.
UpgradeStepInProgress	An upgrade is in progress.
UpgradeQueued	Waiting for other control planes in the Space to finish upgrading.
UpgradePaused	Upgrades are paused. The message says where the pause is set.
UpgradeHalted	An upgrade failed and needs you to resume it.
UpgradeBlocked	The control plane can't upgrade. The message says why.

Metrics

Each control plane exposes these metrics:

Metric	Description
spaces_control_plane_kubernetes_version	The current Kubernetes version, in the version label.
spaces_control_plane_kubernetes_upgrade_queued	1 while the upgrade is queued.
spaces_control_plane_kubernetes_upgrade_paused	1 while upgrades are paused.
spaces_control_plane_kubernetes_upgrade_halted	1 while the upgrade is halted.

To collect them, scrape the metrics port of pods labeled app.kubernetes.io/name: mxp-controller.

Resume a halted upgrade

An upgrade halts if the control plane isn't healthy within 30 minutes. Halted upgrades don't roll back or retry on their own.

1. Read the KubernetesUpgrade condition message to find the failure. For details, read the mxp-controller logs in the control plane's host namespace on the host cluster:

   kubectl logs -n <host-namespace> deploy/mxp-controller

2. Fix the failure.

3. Set the resume annotation to a new value.

   kubectl annotate controlplane <name> -n <group> \
     spaces.upbound.io/kubernetes-upgrade-resume="$(date -u +%s)" \
     --overwrite

   The condition changes to UpgradeStepInProgress. If upgrades are paused, or the control plane is waiting for other control planes to finish, the condition stays UpgradeHalted and its message says what the resume waits for.