Distribute manifests to control planes

important

ResourceDistribution is an alpha feature in Spaces v1.19 and later, and is off by default.

A ResourceDistribution applies a set of Kubernetes manifests to every control plane it selects in a group. Use it to install the same packages, DeploymentRuntimeConfigs, ProviderConfigs, EnvironmentConfigs, and RBAC in every new control plane.

Enable the feature

Enable the feature when you install or upgrade Spaces:

helm -n upbound-system upgrade spaces \
  oci://xpkg.upbound.io/spaces-artifacts/spaces \
  --reuse-values \
  --set "features.alpha.resourceDistribution.enabled=true" \
  --wait

Permissions

Group admins can create, update, and delete ResourceDistributions. Group editors and viewers can read them.

A distribution applies its manifests as control plane admin, so anyone who can write one, including a GitOps tool, has control plane admin access to every control plane it selects. Grant write access with the same care.

Create a distribution

Create the ResourceDistribution in the group namespace of the control planes it targets. This example installs a Configuration and creates a namespace in every control plane labeled env: prod:

apiVersion: spaces.upbound.io/v1alpha1
kind: ResourceDistribution
metadata:
  name: baseline
  namespace: default
spec:
  controlPlaneSelector:
    labelSelectors:
    - matchLabels:
        env: prod
  sources:
  - type: Inline
    inline:
      manifests:
      - apiVersion: pkg.crossplane.io/v1
        kind: Configuration
        metadata:
          name: configuration-aws-network
        spec:
          package: xpkg.upbound.io/upbound/configuration-aws-network:v2.2.0
      - apiVersion: v1
        kind: Namespace
        metadata:
          name: team-a

Spaces applies the manifests in the order listed and labels each object with resourcedistribution.spaces.upbound.io/managed-by: baseline. A manifest that depends on something not installed yet, such as a ProviderConfig whose Provider is still installing, is retried until it applies.

Don't put Secrets or credentials in a distribution. Group viewers can read its manifests. Use shared secrets instead.

Select control planes

Use labelSelectors to match control planes by label, or names to list them:

controlPlaneSelector:
  names:
  - ctp-a
  - ctp-b

An empty label selector matches every control plane in the group:

controlPlaneSelector:
  labelSelectors:
  - {}

New control planes, and control planes whose labels start to match, get the manifests automatically. A control plane can match more than one distribution.

Check status

kubectl get resourcedistribution baseline -o wide

NAME       READY   PROVISIONED   MESSAGE   AGE
baseline   True    2/2                     1m

PROVISIONED counts the control planes that applied the latest version. When a control plane fails, MESSAGE names it and shows the error. READY means the manifests applied, so check the objects in the control plane to confirm a package is healthy.

For the full result in one control plane, look at its ResourceDistributionBinding:

kubectl get resourcedistributionbindings -l spaces.upbound.io/resource-distribution=baseline

Update a distribution

Edit and apply the distribution. Spaces updates every selected control plane at once, and the rollout is done when READY is True and PROVISIONED shows every control plane. To roll back, apply the previous version.

By default, Spaces keeps each object matching its manifest and recreates it if someone deletes it. To create an object once and leave it to another tool, put it in a source with strategy: ApplyOnce:

sources:
- type: Inline
  strategy: ApplyOnce
  inline:
    manifests:
    - apiVersion: pkg.crossplane.io/v1
      kind: Configuration
      metadata:
        name: configuration-aws-network
      spec:
        package: xpkg.upbound.io/upbound/configuration-aws-network:v2.2.0

Adopt existing objects

A distribution only manages objects labeled with its name, and reports a conflict for any other object with the same name. To adopt an object, or to move objects after you rename a distribution, set the label in the control plane:

kubectl label <kind> <name> [-n <namespace>] resourcedistribution.spaces.upbound.io/managed-by=<distribution-name>

Delete objects

By default, deletionPolicy is Orphan, and Spaces never deletes applied objects. Set it to Delete to delete an object when you remove its manifest, and every applied object when you delete the distribution:

spec:
  deletionPolicy: Delete

Deleting a Namespace deletes everything in it, and deleting a package deletes the APIs it installed. Spaces never deletes objects in control planes the distribution stops selecting, objects created with ApplyOnce, or objects with this annotation:

metadata:
  annotations:
    resourcedistribution.spaces.upbound.io/prune: "false"

A distribution with deletionPolicy: Delete waits for every control plane to delete its objects. If a control plane can't prune its resources, set the policy to Orphan to finish the deletion:

kubectl patch resourcedistribution baseline --type merge -p '{"spec":{"deletionPolicy":"Orphan"}}'

Disable the feature

Delete every ResourceDistribution before you turn the feature off. To keep their objects, set deletionPolicy: Orphan first.