Deploy a monitoring stack Commercial Edition

Plane exports traces, metrics and logs over OTLP, but it does not ship a backend to store or view them. This guide deploys one next to Plane on the same Kubernetes cluster, using the upstream Helm charts for the OpenTelemetry Collector, Prometheus, Grafana, Loki, Tempo and Alloy. At the end you can open Grafana and see request latency per API route, read a worker's logs, and jump from a log line to the trace that produced it.

If you already run Prometheus, Grafana Cloud, Datadog or another OTLP-compatible backend, skip this page and follow Configure OpenTelemetry instead. For the list of metric names, log fields and span names Plane emits, see Metrics, logs and traces.

Scope

This guide covers Kubernetes deployments using the plane-enterprise Helm chart version 3.3.0 or later. The values below give you a working single-replica stack with local disks. They are a starting point, not a hardened production install. See Before production for what to change.

How the pieces fit

Every Plane service sends to one address, the collector. The collector exposes metrics for Prometheus to scrape, forwards traces to Tempo and forwards logs to Loki. Only the Plane AI services export logs over OTLP. Everything else writes JSON logs to stdout, so an Alloy DaemonSet tails the container logs and pushes them to Loki, and skips the Plane AI containers to avoid storing every line twice.

The charts and versions this guide was written against:

Component	Helm chart	Chart version	Role
OpenTelemetry Collector	open-telemetry/opentelemetry-collector	0.178.1	Receives OTLP from Plane, fans out to the three stores
Prometheus and Grafana	prometheus-community/kube-prometheus-stack	92.16.1	Metrics storage, dashboards, node and pod metrics
Tempo	grafana-community/tempo	3.2.1	Trace storage, service graph and span metrics
Loki	grafana-community/loki	20.2.0	Log storage
Alloy	grafana/alloy	1.15.0	Tails container logs from the Plane namespace

The Tempo and Loki charts moved from the grafana repository to grafana-community in January 2026. The grafana copies still install but no longer receive updates. Newer chart versions usually work with the same values. If a key moves, the chart's own values.yaml is the reference.

The whole stack needs roughly 2 CPU and 6 GiB of memory at idle, most of it in Prometheus and Loki. Budget 60 GiB of persistent storage for the retention periods used below.

Before you begin

• A Kubernetes cluster with Plane installed from the plane-enterprise chart. The examples use release name plane-app in namespace plane. Replace both if yours differ.
• helm 3.x and kubectl with cluster-admin access.
• A default StorageClass, so the PersistentVolumeClaims below bind on their own.

Add the chart repositories and create the namespace:

helm repo add open-telemetry https://open-telemetry.github.io/opentelemetry-helm-charts
helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm repo add grafana-community https://grafana-community.github.io/helm-charts
helm repo add grafana https://grafana.github.io/helm-charts
helm repo update

kubectl create namespace observability

Install order matters only for Tempo, which remote-writes span metrics to Prometheus. Install Prometheus first and the rest in any order.

Step 1: Prometheus and Grafana

kube-prometheus-stack installs Prometheus, Grafana, Alertmanager, the Prometheus Operator, kube-state-metrics and node-exporter. Save this as kps-values.yaml:

fullnameOverride: kps

prometheus:
  prometheusSpec:
    retention: 15d
    # Tempo's metrics generator writes span metrics here.
    enableRemoteWriteReceiver: true
    storageSpec:
      volumeClaimTemplate:
        spec:
          resources:
            requests:
              storage: 30Gi
    additionalScrapeConfigs:
      - job_name: plane
        scrape_interval: 15s
        static_configs:
          - targets: ["otel-collector.observability.svc:8889"]

grafana:
  adminPassword: change-me
  additionalDataSources:
    - name: Loki
      type: loki
      uid: loki
      access: proxy
      url: http://loki.observability.svc:3100
      jsonData:
        maxLines: 1000
        derivedFields:
          # Turns the trace_id in a log line into a link to the trace in Tempo.
          - name: TraceID
            datasourceUid: tempo
            matcherRegex: '"trace_id":\s*"(\w+)"'
            url: "$${__value.raw}"
    - name: Tempo
      type: tempo
      uid: tempo
      access: proxy
      url: http://tempo.observability.svc:3200
      jsonData:
        tracesToLogsV2:
          datasourceUid: loki
          spanStartTimeShift: "-5m"
          spanEndTimeShift: "5m"
          filterByTraceID: true
          tags:
            - key: service.name
              value: service_name
        tracesToMetrics:
          datasourceUid: prometheus
          tags:
            - key: service.name
              value: service_name
        serviceMap:
          datasourceUid: prometheus
        nodeGraph:
          enabled: true

The plane scrape job reads the collector's Prometheus endpoint. Every application metric Plane emits arrives through it. Node and pod metrics come from the chart's own kube-state-metrics and node-exporter, so you get CPU, memory and restart counts for the Plane pods without extra setup.

Install it:

helm install kps prometheus-community/kube-prometheus-stack \
  --namespace observability \
  --version 92.16.1 \
  -f kps-values.yaml

The built-in Prometheus data source gets the uid prometheus, which the Tempo data source above relies on.

Cluster already running a Prometheus Operator

If another Prometheus Operator already runs in the cluster, for example from Rancher Monitoring or an earlier kube-prometheus-stack, do not install a second one. Two operators reconcile the same Prometheus objects and fight over them, and a second node-exporter cannot bind host port 9100. Save this as kps-existing-operator.yaml and pass it as a second -f on the install:

crds:
  enabled: false
prometheusOperator:
  enabled: false
nodeExporter:
  enabled: false
# Operators older than v0.79 cannot run Prometheus 3. Pin the 2.x line.
prometheus:
  prometheusSpec:
    image:
      tag: v2.55.1

The existing operator then manages the Prometheus this chart creates. Check its version with kubectl get deploy -A -o wide | grep prometheus-operator. If it is v0.79 or newer, drop the image.tag pin. The ServiceMonitor and PrometheusRule objects still select on the release: kps label, so nothing else changes.

Expose Grafana through an Ingress

Port-forwarding is enough while you set things up. To give your team a URL, add an Ingress for Grafana with the same ingress class and cert-manager issuer you use for Plane. Save this as kps-ingress.yaml and pass it as another -f on the install or upgrade:

grafana:
  grafana.ini:
    server:
      root_url: https://grafana.example.com
  ingress:
    enabled: true
    ingressClassName: nginx
    annotations:
      cert-manager.io/cluster-issuer: letsencrypt-prod
    hosts:
      - grafana.example.com
    tls:
      - secretName: grafana-tls
        hosts:
          - grafana.example.com

Replace the host, the ingress class and the issuer name. If you do not run cert-manager, create the grafana-tls Secret from your own certificate and drop the annotation. Grafana has its own login, so this is safe to expose once adminPassword is no longer the placeholder.

helm upgrade kps prometheus-community/kube-prometheus-stack \
  --namespace observability \
  --version 92.16.1 \
  -f kps-values.yaml -f kps-ingress.yaml

Prometheus, Alertmanager, Loki and Tempo have no login

Grafana is the only component built to face users. The others accept any request they receive. Keep them inside the cluster and reach them with kubectl port-forward when you need the raw UI. If you must publish Prometheus, put authentication in front of it. With ingress-nginx, a basic-auth Secret does the job:

htpasswd -c auth admin
kubectl create secret generic prometheus-basic-auth --namespace observability --from-file=auth

prometheus:
  ingress:
    enabled: true
    ingressClassName: nginx
    annotations:
      cert-manager.io/cluster-issuer: letsencrypt-prod
      nginx.ingress.kubernetes.io/auth-type: basic
      nginx.ingress.kubernetes.io/auth-secret: prometheus-basic-auth
      nginx.ingress.kubernetes.io/auth-realm: "Prometheus"
    hosts:
      - prometheus.example.com
    tls:
      - secretName: prometheus-tls
        hosts:
          - prometheus.example.com

Alertmanager takes the same block under alertmanager.ingress.

Step 2: Tempo

Save this as tempo-values.yaml:

tempo:
  reportingEnabled: false
  retention: 48h
  metricsGenerator:
    enabled: true
    remoteWriteUrl: http://kps-prometheus.observability.svc:9090/api/v1/write
  overrides:
    defaults:
      metrics_generator:
        processors:
          - service-graphs
          - span-metrics

persistence:
  enabled: true
  size: 20Gi

The metrics generator turns every span into traces_spanmetrics_* series in Prometheus. That gives you latency and error rates for Celery tasks and for the Node services from the traces alone, and it feeds the service map in Grafana.

helm install tempo grafana-community/tempo \
  --namespace observability \
  --version 3.2.1 \
  -f tempo-values.yaml

Step 3: Loki

Save this as loki-values.yaml:

deploymentMode: Monolithic

loki:
  auth_enabled: false
  commonConfig:
    replication_factor: 1
  storage:
    type: filesystem
  schemaConfig:
    configs:
      - from: "2024-01-01"
        store: tsdb
        object_store: filesystem
        schema: v13
        index:
          prefix: loki_index_
          period: 24h
  limits_config:
    retention_period: 168h
    allow_structured_metadata: true
  compactor:
    retention_enabled: true
    delete_request_store: filesystem

singleBinary:
  replicas: 1
  persistence:
    size: 20Gi

read:
  replicas: 0
write:
  replicas: 0
backend:
  replicas: 0

gateway:
  enabled: false
chunksCache:
  enabled: false
resultsCache:
  enabled: false
lokiCanary:
  enabled: false
test:
  enabled: false

Loki 3 accepts OTLP logs on /otlp and turns the service.name, k8s.namespace.name and deployment.environment.name resource attributes into index labels by default. The other attributes land in structured metadata, which LogQL can still filter on.

helm install loki grafana-community/loki \
  --namespace observability \
  --version 20.2.0 \
  -f loki-values.yaml

Step 4: OpenTelemetry Collector

The collector is the only component Plane talks to. Save this as otel-values.yaml:

fullnameOverride: otel-collector
mode: deployment
replicaCount: 1

image:
  repository: otel/opentelemetry-collector-contrib

resources:
  limits:
    memory: 1Gi

ports:
  prometheus:
    enabled: true
    containerPort: 8889
    servicePort: 8889
    protocol: TCP
  jaeger-compact:
    enabled: false
  jaeger-thrift:
    enabled: false
  jaeger-grpc:
    enabled: false
  zipkin:
    enabled: false

config:
  receivers:
    otlp:
      protocols:
        grpc:
          endpoint: ${env:MY_POD_IP}:4317
        http:
          endpoint: ${env:MY_POD_IP}:4318
          # Only needed if you turn on browser tracing.
          cors:
            allowed_origins: ["https://plane.example.com"]
            allowed_headers: ["*"]

  processors:
    batch:
      timeout: 10s
      send_batch_size: 1024
    # Plane sets service.name per workload. This only fills the gap when a sender has none.
    resource:
      attributes:
        - key: service.name
          action: insert
          value: plane
    # Per-process attributes would become labels on every metric series. Drop them before export.
    resource/metrics:
      attributes:
        - key: service.instance.id
          action: delete
        - key: process.pid
          action: delete
        - key: process.command_args
          action: delete
        - key: process.command_line
          action: delete
        - key: process.executable.path
          action: delete
        - key: process.executable.name
          action: delete
        - key: process.owner
          action: delete
        - key: process.runtime.description
          action: delete

  exporters:
    prometheus:
      endpoint: ${env:MY_POD_IP}:8889
      resource_to_telemetry_conversion:
        enabled: true
    otlp_grpc/tempo:
      endpoint: tempo.observability.svc:4317
      tls:
        insecure: true
    otlp_http/loki:
      endpoint: http://loki.observability.svc:3100/otlp

  service:
    pipelines:
      traces:
        receivers: [otlp]
        processors: [memory_limiter, resource, batch]
        exporters: [otlp_grpc/tempo]
      metrics:
        receivers: [otlp]
        processors: [memory_limiter, resource, resource/metrics, batch]
        exporters: [prometheus]
      logs:
        receivers: [otlp]
        processors: [memory_limiter, resource, batch]
        exporters: [otlp_http/loki]

Three choices in this file matter later:

• resource_to_telemetry_conversion copies resource attributes such as service.name onto every metric as labels. The queries on the reference page filter on service_name="api" because of this.
• The resource/metrics processor keeps per-process attributes off the metric labels. Without it, each worker restart creates a new set of series.
• The CORS block on the HTTP receiver is only used by browser tracing. Set it to your Plane domain or remove the block.

The chart merges this config with its defaults, which already define the memory_limiter processor and the health_check extension.

helm install otel-collector open-telemetry/opentelemetry-collector \
  --namespace observability \
  --version 0.178.1 \
  -f otel-values.yaml

Step 5: Alloy for container logs

Alloy reads container logs through the Kubernetes API, so it needs no host path mounts. Save this as alloy-values.yaml:

alloy:
  configMap:
    content: |
      discovery.kubernetes "plane" {
        role = "pod"
        namespaces {
          names = ["plane"]
        }
      }

      discovery.relabel "plane" {
        targets = discovery.kubernetes.plane.targets

        rule {
          source_labels = ["__meta_kubernetes_namespace"]
          target_label  = "namespace"
        }
        rule {
          source_labels = ["__meta_kubernetes_pod_name"]
          target_label  = "pod"
        }
        rule {
          source_labels = ["__meta_kubernetes_pod_container_name"]
          target_label  = "container"
        }
        // Container names carry the release prefix (plane-app-api). Strip it so
        // service_name matches the OTEL_SERVICE_NAME the chart gives each workload.
        rule {
          source_labels = ["__meta_kubernetes_pod_container_name"]
          regex         = "plane-app-(.+)"
          target_label  = "service_name"
          replacement   = "$1"
        }
        // The Plane AI services ship their logs over OTLP. Skip their stdout.
        rule {
          source_labels = ["__meta_kubernetes_pod_container_name"]
          regex         = "plane-app-(pi-api|pi-worker|pi-beat)"
          action        = "drop"
        }
      }

      loki.source.kubernetes "plane" {
        targets    = discovery.relabel.plane.output
        forward_to = [loki.write.default.receiver]
      }

      loki.write "default" {
        endpoint {
          url = "http://loki.observability.svc:3100/loki/api/v1/push"
        }
      }

Replace plane-app in both regular expressions with your Plane release name, and plane in names with your Plane namespace. The chart names every container <release>-<service>, which is what the first regular expression strips. Confirm the names on your cluster before you rely on the rule:

kubectl get pods -n plane -o jsonpath='{range .items[*]}{.spec.containers[*].name}{"\n"}{end}' | sort -u

To cover several Plane instances with one Alloy, list every namespace in names and widen the regular expressions, for example (plane-app|plane-staging)-(.+) with replacement = "$2".

helm install alloy grafana/alloy \
  --namespace observability \
  --version 1.15.0 \
  -f alloy-values.yaml

Step 6: Point Plane at the collector

Turn on OpenTelemetry in the plane-enterprise release:

helm upgrade plane-app plane/plane-enterprise \
  --namespace plane \
  --reuse-values \
  --set observability.otel.enabled=true \
  --set observability.otel.endpoint=http://otel-collector.observability.svc:4318 \
  --set observability.otel.protocol=http/protobuf \
  --set observability.otel.environment=production \
  --set observability.otel.resourceAttributes="k8s.namespace.name=plane"

Or in your values file:

observability:
  otel:
    enabled: true
    endpoint: http://otel-collector.observability.svc:4318
    protocol: http/protobuf
    environment: production
    resourceAttributes: "k8s.namespace.name=plane"

Use the HTTP port and http/protobuf rather than gRPC on 4317. The Plane AI services and the space server export over OTLP/HTTP only, and one endpoint for every service keeps the configuration short. The k8s.namespace.name attribute becomes a label in Loki and Prometheus, so a cluster that runs several Plane instances can tell them apart.

The chart's default sampler is always_on. That is right for the first days. Once you have confirmed the stack works, see Sampling to keep a share of traces on a busy instance.

To add browser traces and page-load timings from users' browsers, expose the collector's 4318 port through an Ingress with TLS and set observability.otel.frontend.enabled and observability.otel.frontend.endpoint. The CORS block in the collector values must then allow your Plane origin. Browser tracing lists the constraints.

Verify the stack

1. Every pod in observability is Running:

   kubectl get pods -n observability

2. The collector is receiving data. The collector binds its ports to the pod IP rather than to localhost, so kubectl port-forward to it does not work. Read its Prometheus endpoint from inside the cluster instead:

   kubectl run otel-curl --rm -i --restart=Never -n observability --image=curlimages/curl:8.11.1 \
     -- sh -c 'curl -s otel-collector.observability.svc:8889/metrics | grep -c http_server_duration_milliseconds_bucket'

   A non-zero count means the API is exporting. If it stays at zero, check the API pod's environment with kubectl exec deploy/plane-app-api -n plane -- env | grep OTEL_.

3. Prometheus scrapes the collector. Open http://localhost:9090/targets after kubectl port-forward -n observability svc/kps-prometheus 9090:9090 and confirm the plane job is UP.

4. Open Grafana with kubectl port-forward -n observability svc/kps-grafana 3000:80, log in as admin with the password from kps-values.yaml, and run one query per data source in Explore:

   Data source	Query	Expected result
   Prometheus	sum by (service_name) (rate(http_server_duration_milliseconds_count[5m]))	One series per Plane service
   Loki	{service_name="api"} | json | name="plane.api.request"	One line per API request
   Tempo	{resource.service.name="api"}	Recent API traces

5. Open any API log line in Loki. The TraceID link next to trace_id opens the trace in Tempo, and the Logs for this span button in Tempo brings you back.

The queries worth putting on a dashboard are on the Metrics, logs and traces page.

Before production

The values above favour a short path to a working stack. Change these before you depend on it:

• Storage. Prometheus, Loki and Tempo each use a single PersistentVolume. For anything beyond one small cluster, move Loki and Tempo to object storage (S3, GCS, Azure Blob, or MinIO) through their storage values, and consider Prometheus remote-write to a long-term store.
• Grafana access. Replace adminPassword, put Grafana behind an Ingress with TLS as shown in Expose Grafana through an Ingress, and configure an identity provider through grafana.grafana.ini. Do not expose Prometheus, Loki or Tempo outside the cluster without authentication in front of them.
• Node consolidation. Karpenter and the cluster autoscaler evict pods from underused nodes. When that hits Prometheus, Loki or Tempo, the pod waits for its volume to detach from the old node and reattach, which takes minutes and leaves gaps in the data. Annotate those pods with karpenter.sh/do-not-disrupt: "true" through each chart's podAnnotations, or pin them to a node group that is not consolidated.
• Network policy. Only the Plane namespace needs to reach the collector on 4317 and 4318, and only the collector needs to reach Loki and Tempo.
• Retention and sampling. Trace volume grows with request volume. Lower observability.otel.samplerArg before raising Tempo's retention.
• Alerting. kube-prometheus-stack installs Alertmanager. Add a receiver under alertmanager.config and load the starter alert rules.
• Managed backends. The collector is the only component Plane depends on. To use Amazon Managed Service for Prometheus, Google Cloud Managed Service for Prometheus, Azure Monitor or Grafana Cloud instead of the self-hosted stores, replace the exporters in otel-values.yaml and leave the Plane configuration unchanged.

Metrics, logs and traces

Everything Plane emits, with the queries to graph and debug it.

Configure OpenTelemetry

Every chart value, authentication headers, sampling and browser tracing.