helm -n upbound-system upgrade --install spaces \
  oci://xpkg.upbound.io/spaces-artifacts/spaces \
  --version "${SPACES_VERSION}" \
  --set "ingress.host=${SPACES_ROUTER_HOST}" \
  --set "clusterType=${SPACES_CLUSTER_TYPE}" \
  --set "account=${UPBOUND_ACCOUNT}" \
  --set "features.alpha.observability.enabled=true" \
  --wait

SharedTelemetryConfig

SharedTelemetryConfig is a custom resource that defines the telemetry configuration for a group of control planes. This resources allows you to specify the exporters and pipelines your control planes use to send telemetry data to your external observability backends.

The following is an example of a SharedTelemetryConfig resource that sends metrics and traces to New Relic:

apiVersion: observability.spaces.upbound.io/v1alpha1
kind: SharedTelemetryConfig
metadata:
  name: newrelic
  namespace: default
spec:
  controlPlaneSelector:
    labelSelectors:
      - matchLabels:
          org: foo
  exporters:
    otlphttp:
      endpoint: https://otlp.nr-data.net
      headers:
        api-key: YOUR_API_KEY
  exportPipeline:
    metrics: [otlphttp]
    traces: [otlphttp]
    logs: [otlphttp]

The controlPlaneSelector field specifies the control planes that use this configuration. The exporters field specifies the configuration for the exporters. Each exporter configuration is unique and corresponds to its OpenTelemetry Collector configuration. The exportPipeline field specifies the control plane pipelines that send telemetry data to the exporters. The metrics, traces and logs fields specify the names of the pipelines the control planes use to send metrics, traces, and logs respectively. The names of the pipelines correspond to the exporters in the OpenTelemetry Collector service pipeline configuration.

Usage

In a Space, you can configure per control plane group telemetry settings by creating one or more SharedTelemetryConfig resources.

Currently supported exporters are:

• datadog (review the OpenTelemetry documentation for configuration details)
• otelhttp (used by New Relic among others, review the New Relic documentation for configuration details)

The example below shows how to configure a SharedTelemetryConfig resource to send metrics, traces and logs to Datadog:

apiVersion: observability.spaces.upbound.io/v1alpha1
kind: SharedTelemetryConfig
metadata:
  name: datadog
  namespace: default
spec:
  controlPlaneSelector:
    labelSelectors:
      - matchLabels:
          org: foo
  exporters:
    datadog:
      api:
        site: ${DATADOG_SITE}
        key: ${DATADOG_API_KEY}
  exportPipeline:
    metrics: [datadog]
    traces: [datadog]
    logs: [datadog]

Status

If successful, Upbound creates the SharedTelemetryConfig resource and provisions the OpenTelemetry Collector for the selected control plane. To see the status, run kubectl get stc:

 kubectl get stc
NAME       SELECTED   FAILED   PROVISIONED   AGE
datadog    1          0        1             63s

• SELECTED shows the number of control planes selected by the SharedTelemetryConfig.
• FAILED shows the number of control planes that failed to provision the OpenTelemetry Collector.
• PROVISIONED shows the provisioned and running OpenTelemetry Collectors on each control plane.

To return the names of control planes selected and provisioned, review the resource status:

...
status:
  selected:
    - ctp
  provisioned:
    - ctp

If a conflict or another issue occurs, the failed control planes status returns the failure conditions:

k get stc
NAME       SELECTED   FAILED   PROVISIONED   AGE
datadog    1          1        0             63s

...
status:
  failed:
  - conditions:
    - lastTransitionTime: "2024-04-26T09:32:28Z"
      message: 'control plane dev is already managed by another SharedTelemetryConfig:
        newrelic'
      reason: SelectorConflict
      status: "True"
      type: Failed
    controlPlane: ctp
  selectedControlPlanes:
  - ctp

Upbound marks the control plane as provisioned only if the OpenTelemetry Collector is deployed and running. There could be a delay in the status update if the OpenTelemetry Collector is currently deploying:

 k get stc
NAME       SELECTED   FAILED   PROVISIONED   AGE
datadog    1          0        0             63s

Space-level observability

Observability is available in preview at the Space level. This feature allows you to observe your Space infrastructure. To enable this feature, set the features.alpha.observability.enabled flag to true when installing Spaces.

When you enable observability in a Space, Upbound deploys a single OpenTelemetry Collector to collect and export metrics and logs to your configured observability backends.

To configure how Upbound exports, review the spacesCollector value in your Space installation Helm chart. Below is an example of an otlphttp compatible endpoint.

observability:
  spacesCollector:
    config:
      exporters:
        otlphttp:
          endpoint: "<your-endpoint>"
          headers:
            api-key: YOUR_API_KEY
      exportPipeline:
        logs: 
          - otlphttp
        metrics: 
          - otlphttp

You can export metrics and logs from your Crossplane installation, Spaces infrastructure (controller, API, router, etc.), provider-helm, and provider-kubernetes.

OpenTelemetryCollector image

Control plane (SharedTelemetry) and Space observability deploy the same custom OpenTelemetry Collector image. The OpenTelemetry Collector image supports otlhttp, datadog, and debug exporters. For more information on observability configuration, review the Helm chart reference.