Query metrics with the API

Use the Insights API to query Crossplane metrics from every control plane at once, then follow a problem from the fleet down to the resources that cause it.

Before you begin

• A hub kubectl context. See Configure kubectl for the Hub.
• Permission to read resources on at least one control plane. Queries only cover control planes you have permission to read. See Access-scoped results.
• Your platform team turns on Metrics and sets up the pipeline. See Metrics pipeline.
• jq.

Run a query

A query is a Query in the fleet.hub.upbound.io/v1alpha1 API group. Create one with a PromQL expression and a time range, and Hub returns the results in the same object. Hub doesn't store queries, so they have no name.

kubectl --context=hub create -o json -f - <<'EOF' | jq '.results.series'
apiVersion: fleet.hub.upbound.io/v1alpha1
kind: Query
query:
  promql: 'sum(rate(controller_runtime_reconcile_total[5m])) by (control_plane_name, result)'
  start: "2026-11-06T12:00:00Z"
  end: "2026-11-06T13:00:00Z"
  step: 5m
EOF

Field	Required	Description
query.promql	Yes	The PromQL expression.
query.start	Yes	The start of the range, as an RFC3339 timestamp such as 2026-11-06T12:00:00Z.
query.end	Yes	The end of the range, after start.
query.step	Yes	The resolution, as a duration such as 15s, 5m, or 1h30m. Days and weeks, such as 1d, aren't accepted.

Each entry in results.series has the series' labels, and its values as timestamp, in Unix seconds, and value pairs:

[
  {
    "labels": { "control_plane_name": "webapp-us-west-1", "result": "success" },
    "values": [
      { "timestamp": 1793966400, "value": 1.01 },
      { "timestamp": 1793966700, "value": 2.02 }
    ]
  }
]

Filter and group by control_plane_name and realm. See How series identify their control plane.

Connect metrics to resources

Metrics and the resource list describe the same objects. Two metric families lead back to resources:

Metric	Leads to	Through
kube_customresource_resource_condition	One resource	The group, kind, name, and namespace labels, which match the resource in the resource list.
controller_runtime_*	A kind of resource	The controller label, which names the controller for that kind.

kube_customresource_resource_condition has one series per condition per resource. Its condition_type label names the condition, and its value is 1 when the condition is True, or 0 when it's False or Unknown.

Crossplane names each controller after what it reconciles:

Controller	Reconciles	Example
composite/<plural>.<group>	Composite resources of an XRD	composite/webapps.platform.example.com
claim/<plural>.<group>	Claims of an XRD, named after the XRD	claim/xdatabases.platform.example.com
managed/<kind>.<group>	Managed resources of a kind, in lowercase	managed/bucket.s3.aws.upbound.io
packages/<kind>.pkg.crossplane.io	Packages and their revisions, in lowercase	packages/providerrevision.pkg.crossplane.io

Other controllers exist, so list the real values before you match on them:

topk(20, sum(increase(controller_runtime_reconcile_total[15m])) by (controller, result))

Follow a problem from the fleet to a resource

1. Find the control planes with the most reconcile errors:

   topk(5, sum(rate(controller_runtime_reconcile_errors_total[5m])) by (control_plane_name))

2. On the worst one, find which controller, and so which kind of resource, the errors come from:

   topk(5, sum(rate(controller_runtime_reconcile_errors_total{control_plane_name="webapp-us-west-1"}[5m])) by (controller))

3. Find the resources of that kind that aren't ready:

   kube_customresource_resource_condition{control_plane_name="webapp-us-west-1", kind="WebApp", condition_type="Ready"} == 0

4. Open one of them in the resource list for its conditions, events, and YAML. See Query resources with the API.

To find resources that flap, use changes() over the same condition series.

Troubleshoot

Status	Cause
404	Your platform team hasn't turned on Metrics.
400	The PromQL doesn't parse, refers to the control_plane_id label, or the backend rejected the query. The message says which.
403	You don't have permission to read any control plane.
422	A required field is missing or malformed, or end isn't after start.
500	Hub couldn't reach the backend, or the query timed out.
Empty series	No control plane you can read sent a matching series in the range.

Next steps

• Metrics
• Query resources with the API
• Metrics pipeline