How to get metrics and logs from Kubernetes to SysEleven Observability

In the following steps, you will learn how to set up the k8s-monitoring Helm chart and connect it to SysEleven Observability.

Requirements

You require access to the following resources:

• Access to a Kubernetes cluster via kubectl
• Permissions to create Service Accounts for your SysEleven organization/project.
• Helm needs to be installed.

Create Service Account

1. Log in to https://dashboard.syseleven.de

2. Create an Service Account Click on the user avatar in the top‑right corner → User Management → SERVICE ACCOUNTS → CREATE SERVICE ACCOUNT

3. Click on + CREATE in the CREDENTIALS section and save the secret in your password manager.

4. Select the service account → Go to the tab PROJECTS → + GRANT → Select your project and set the following permissions:

• Write Logs
• Write Metrics
• Write Rules

Installing the k8s-monitoring Helm chart

To stream all metrics and logs into SysEleven Observability we deploy the k8s‑monitoring Helm chart.

1. Configure the Helm values in a values.yaml file. Make sure that you have the correct organization ID, project ID, and bearer token configured.

values.yaml file

destinations:
  Metrics:
    type: prometheus
    url: https://metrics.syseleven.services/<organization id>/<project id>/api/v1/push
    auth:
      type: bearerToken
      bearerToken: <Service Account Secret>
  Logs:
    type: loki
    url: https://logs.syseleven.services/<organization id>/<project id>/loki/api/v1/push
    auth:
      type: bearerToken
      bearerToken: <Service Account Secret>
collectors:
  alloy-metrics:
    presets:
      - clustered
      - statefulset
  alloy-logs:
    presets:
      - filesystem-log-reader
      - daemonset
  alloy-singleton:
    presets:
      - singleton
clusterMetrics:
  enabled: true
  collector: alloy-metrics
telemetryServices:
  kube-state-metrics:
    deploy: true
cluster:
  name: awesome-project-dev
clusterEvents:
  enabled: true
  collector: alloy-singleton
nodeLogs:
  enabled: true
  collector: alloy-logs
podLogsViaLoki:
  enabled: true
  collector: alloy-logs
prometheusOperatorObjects:
  enabled: true
  collector: alloy-metrics
integrations:
  alloy:
    instances:
      - name: alloy
        labelSelectors:
          app.kubernetes.io/name:
            - alloy-metrics
            - alloy-logs
            - alloy-singletonalloy-logs
  collector: alloy-metrics

2. Install the Helm chart

Info

Newer versions of the k8s-monitoring Helm chart should also work. For the latest available version and upstream guidance, please refer to the upstream project and its release information.

# Install Prometheus Operator Object CRDs (ServiceMonitor, PodMonitor, Probe)
helm upgrade --install --version 28.0.1 \
  prometheus-operator-crds oci://ghcr.io/prometheus-community/charts/prometheus-operator-crds

helm upgrade --install --version 4.0.3 \
  --values values.yaml \
  --create-namespace --namespace k8s-monitoring \
  k8s-monitoring oci://ghcr.io/grafana/helm-charts/k8s-monitoring

Configuring in Grafana

To visualize the data, you must add the data sources to Grafana. To do this, create a new service account with the following permissions:

• Read Logs
• Read Metrics
• Read Alertmanager
• Write Alertmanager

Config FileUI

Data sources can be configured in Grafana via the datasources.yaml in the provisioning/datasources directory.

datasources:
  - name: Metrics
    type: prometheus
    url: https://metrics.syseleven.services/<organization ID>/<project ID>/prometheus
    access: proxy
    isDefault: true
    jsonData:
      prometheusType: Mimir
      httpHeaderName1: Authorization
    secureJsonData:
      httpHeaderValue1: Bearer <Service Account Secret>
  - name: Logs
    type: loki
    url: https://logs.syseleven.services/<organization ID>/<project ID>
    access: proxy
    jsonData:
      httpHeaderName1: Authorization
    secureJsonData:
      httpHeaderValue1: Bearer <Service Account Secret>

1. Log in to Grafana
2. Click on the Grafana logo in the top-left corner → Select Data sources
3. Click on + Add new data source and select Prometheus (metrics)

4. Enter the Prometheus server URL:

   https://metrics.syseleven.services/<organization ID>/<project ID>/prometheus

5. In the section HTTP headers click on + Add another header

Header: Authorization Value: Bearer <Service Account Secret>

For Loki (logs), repeat the steps above, using the Loki data source and the following server URL instead:

https://logs.syseleven.services/<organization ID>/<project ID>