# Loki-Promtail

## Overview

The source code and default configuration of the Building Block is available in our [code.syseleven.de](<https://code.syseleven.de/syseleven/building-blocks/helmfiles/loki-promtail>). Infos on release notes and new features please follow [Release notes loki-promtail](<https://code.syseleven.de/syseleven/building-blocks/helmfiles/loki-promtail/-/blob/main/RELEASE-NOTES.md>)

### Prerequisites On Loki Promtail

Proceed with the following prerequisite description to use the Building Block out of the box.

#### Recommended setup

A recommended resource overview is listed in the table below.

| CPU/vCPU | Memory |
| --- | --- |
| none | 512MiB + 256 per node |

No further activities need to be carried out in advance.

### Adding The Building Block

Add the directory `syseleven-loki-promtail` to your control repository. Add a `.gitlab-ci.yml` to the directory with the following content:

```yaml
include:
  - project: syseleven/building-blocks/helmfiles/loki-promtail
    file: JobDevelopment.yaml
    ref: {{ site.building_blocks.loki_promtail.version|e }}
  - project: syseleven/building-blocks/helmfiles/loki-promtail
    file: JobStaging.yaml
    ref: {{ site.building_blocks.loki_promtail.version|e }}
  - project: syseleven/building-blocks/helmfiles/loki-promtail
    file: JobProduction.yaml
    ref: {{ site.building_blocks.loki_promtail.version|e }}
```

> [!NOTE]
>
> **Info**
>
> Remove environments you are not using by removing their include.

### Required Configuration

No configuration is required.

### What Logs Are Collected?

- The building block consists of two parts: Loki and Promtail. Promtail runs as a DaemonSet on each node in the cluster and collects logs. Loki is running as a central instance in the cluster and stores the logs it receives from Promtail. When looking into the logs, you usually only interact with Loki.
- The building block collects stdout and stderr of all pods in the cluster. In addition the pods deployed by the SysEleven provided building blocks, this also includes the logs of all application pods deployed by the user.
- In addition the pod logs the systemd journals of all nodes in the cluster are collected.
- By default the logs are stored by Loki for one week. The retention period can be adjusted in `values-loki.yaml` [see](<https://code.syseleven.de/syseleven/building-blocks/helmfiles/loki-promtail/-/blob/main/values-loki.yaml>).

### How Can I Access The Logs?

The easiest way to access the logs is [Grafana](<https://grafana.com/docs/loki/latest/getting-started/grafana/>) provided with the kube-stack-prometheus module. We automatically create a Loki data source for you in it, so you can skip this step in the linked upstream documentation. In Grafana, you can use the [explore feature](<https://grafana.com/docs/grafana/latest/explore/>) to view your logs and refine your query. For something more permanent, you can add a [log panel](<https://grafana.com/docs/grafana/latest/panels/visualizations/logs-panel/>) to one of your dashboards.

If you prefer the command line you can use [logcli](<https://grafana.com/docs/loki/latest/query/logcli/>).

### Monitoring

### Additional Alert-Rules

- None

### Additional Grafana Dashboards

- Loki Top 10 producer
- An overview of the top 10 kubernetes namespaces that produce logs
- Loki &amp; Promtail
- An overview of performance metrics from Loki and Promtail

### Scale Loki Volume

To scale the persistent volume Loki stores your data on, perform the following steps:

```sh
# Delete StatefulSet (the PVC will remain)
kubectl delete sts loki

# Patch the PVC to e.g. 10Gi
kubectl patch pvc storage-loki-0 -p '{"spec":{"resources":{"requests":{"storage":"10Gi"}}}}' --type=merge
```

Then, adapt your adapt values.yaml, for example `values-loki-stage.yaml` for the same size as the patch above.

To deploy the StatefulSet again, push your changes and merge them to the default branch - the CI will then deploy it again.

### Scale Setup

This building block consists of multiple components. Each of the components can and must be scaled individually.

### Scaling Loki

Also see [Scaling with Loki](<https://grafana.com/docs/loki/latest/operations/scalability/>) for upstream documentation.

- Scaling replicas is not supported with this building block. See [loki-distributed](<https://github.com/grafana/helm-charts/tree/main/charts/loki-distributed>) for a possible solution.
- Requests/limits for CPU/memory can be adjusted

### Scaling Promtail

- Runs as one DaemonSet on each node, so no further replica scaling is needed
- Requests/limits for CPU/memory can be adjusted

### Migrate From Loki-Promtail To Loki-Alloy

#### Prepare Setup

Delete the clusterrole and clusterrolebinding for loki in your cluster:

```sh
kubectl delete clusterrole loki-clusterrole
kubectl delete clusterrolebinding loki-clusterrolebinding
kubectl delete crd podlogs.monitoring.grafana.com
helm uninstall -n syseleven-loki-promtail promtail
```

#### Install Loki-Alloy

If you have a control-repository with a .gitlab-ci.yml, adjust the package you install:

```yaml
stages:
  - diff
  - deploy

include:
    - /syseleven-ingress-nginx/.gitlab-ci.yml
    - /syseleven-cert-manager/.gitlab-ci.yml
    - /syseleven-external-dns/.gitlab-ci.yml
    - /syseleven-kube-prometheus-stack/.gitlab-ci.yml
#    - /syseleven-loki-promtail/.gitlab-ci.yml # or delete entirely
    - /syseleven-loki-alloy/.gitlab-ci.yml
```

And add a directory called syseleven-loki-alloy with another .gitlab-ci.yml:

```yaml
include:
  # Development environment
  - project: syseleven/building-blocks/helmfiles/loki-alloy
    file: JobDevelopment.yaml
    ref: main
```

Go to your grafana domain, login and go to `Connections > Data Sources`. Click the blue button Add new data source. Choose under `Logging & document databases` loki as source.

Under `Connection` add the URL for the new loki instance. The Default is `http://loki.syseleven-loki-alloy:3100`

#### Cleanup

After the end of your retention period (which defaults to 7 days) you can uninstall `loki-promtail`. To do so, make sure you got the CLI `helm` installed. Afterwards it is safe to remove the namespace.

```sh
helm uninstall -n syseleven-loki-promtail loki loki-promtail-extension event-exporter promtail
kubectl delete namespace syseleven-loki-promtail
```

### Release-Notes

Please find more infos on release notes and new features [release notes Loki-Promtail](<https://code.syseleven.de/syseleven/building-blocks/helmfiles/loki-promtail/-/blob/main/RELEASE-NOTES.md>)
