# OpenSearch

## Overview

The source code and default configuration of the Building Block is available in our [code.syseleven.de](<https://code.syseleven.de/syseleven/building-blocks/helmfiles/opensearch>). Infos on release notes and new features please follow [Release notes OpenSearch](<https://code.syseleven.de/syseleven/building-blocks/helmfiles/opensearch/-/blob/main/RELEASE-NOTES.md>)

## Building Block For OpenSearch

With this Building Block SysEleven provides OpenSearch, an open source search and analytics suite that delivers full-text search, real-time indexing, and advanced analytics for scalable data insights. It consists of

1. The official [OpenSearch Operator](<https://github.com/opensearch-project/opensearch-k8s-operator/tree/main>)
2. A cluster of 3 cluster\_manager/data nodes
3. OpenSearch Dashboards
4. Grafana Dashboards, ServiceMonitor for OpenSearch and PrometheusRules

Kubernetes resources are defined using the official [Helm charts](<https://github.com/opensearch-project/opensearch-k8s-operator/tree/main/charts>).

### Prerequisites

Proceed with the following prerequisite description to use the Building Block out of the box.

#### Recommended setup

For the basic OpenSearch cluster deployed by the Building Block as default a resource overview is listed in the table below.

| CPU/vCPU | Memory |
| --- | --- |
| 5 | 8.5 GB |

Adjust available resources based on the resource requirements of the OpenSearch cluster nodes. For reliability reasons OpenSearch instances of the same kind should not run on the same Kubernetes node.

## Adding The Building Block

Add the directory `opensearch` to your control repository. Add a `.gitlab-ci.yml` to the directory with the following content:

```yaml
include:
  - project: syseleven/building-blocks/helmfiles/opensearch
    file: JobDevelopment.yaml
    ref: { { site.building_blocks.opensearch.version|e } }
  - project: syseleven/building-blocks/helmfiles/opensearch
    file: JobStaging.yaml
    ref: { { site.building_blocks.opensearch.version|e } }
  - project: syseleven/building-blocks/helmfiles/opensearch
    file: JobProduction.yaml
    ref: { { site.building_blocks.opensearch.version|e } }
```

Remove environments you are not using by removing their include.

To choose between major version `2.x` and `3.x` of OpenSearch and OpenSearch Dashboards set the environment variables `OPENSEARCH_MAJOR_VERSION` and `OPENSEARCH_DASHBOARDS_MAJOR_VERSION` to the desired version, the default is 2 for both. This will install the latest version of the chosen branch that the Building Block provides.

## Required Configuration

No configuration is required.

## OpenSearch Cluster Creation

When the Building Block is used with its defaults, it deploys an OpenSearch cluster with three nodes with the role `cluster_manager` and `data`. Each has resource requests of `500m` CPU and `2 GB` memory (request and limit) and 10 GB volume size. As this cluster size is sufficient for general overview and to test the capabilities of the Operator, it is probably undersized. To meet performance requirements a cluster is customized by overwriting the `nodePools` in `values-opensearch-cluster.yaml`

```yaml
cluster:
  nodePools:
    - component: "nodes"
      diskSize: "10Gi"
      replicas: 3
      roles:
        - "cluster_manager"
        - "data"
      resources:
        requests:
          memory: "2Gi"
          cpu: "500m"
        limits:
          memory: "2Gi"
      affinity:
        podAntiAffinity:
          preferredDuringSchedulingIgnoredDuringExecution:
            - weight: 100
              podAffinityTerm:
                labelSelector:
                  matchLabels:
                    opster.io/opensearch-cluster: "opensearch-cluster"
                    opster.io/opensearch-nodepool: "nodes"
                topologyKey: kubernetes.io/hostname
```

## Custom Admin And Dashboards Account (Recommended)

By default the OpenSearch cluster is created with default admin credentials `admin:admin`. To set strong passwords, the `internal_users.yaml` of the Security plugin have to be overwritten to create admin and dashboards accounts. Also, the OpenSearch operator needs to know about those credentials to authenticate API requests there for it is required to prepare secrets containing them. This is done in the values file `values-opensearch-extension.yaml` of the OpenSearch extension chart

```yaml
securityconfig:
  internal_users.yml: |-
    _meta:
      type: "internalusers"
      config_version: 2
    admin:
      hash: "$2a$12$rJeFw143s15v.92cSCPyJu.bhtcym0Q7oY024P8llRIk0sFP0bTwS"
      reserved: true
      backend_roles:
        - "admin"
      description: "Admin user"
    dashboards:
      hash: "$2a$12$wv8p4gfwSruQ1E3qD62kZOqgEutXPDjkIMYvXzYFDJrdW8pZKwK/O"
      reserved: true
      backend_roles:
        - "admin"
      description: "Dashboard user to communicate with OpenSearch"

opensearchAuthSecret:
  admin:
    password: strong-password-here
  dashboards:
    password: another-strong-password-here
```

The Security plugin uses bcrypt hashes which can be created with this command (requires Python with bcrypt module installed)

```bash
python -c 'import bcrypt; print(bcrypt.hashpw("strong-password-here".encode("utf-8"), bcrypt.gensalt(12, prefix=b"2a")).decode("utf-8"))'
```

Now, to actually use the secrets, pass them through the OpenSearch cluster chart's values file `values-opensearch-cluster.yaml`

```yaml
cluster:
  security:
    config:
      adminCredentialsSecret:
        name: "opensearch-auth-admin"
      securityConfigSecret:
        name: "opensearch-securityconfig"

  dashboards:
    opensearchCredentialsSecret:
      name: "opensearch-auth-dashboards"
```

Further information can be found in the official [OpenSearch Operator User Guide](<https://github.com/opensearch-project/opensearch-k8s-operator/blob/main/docs/userguide/main.md#custom-admin-user>)

## Index Snapshots

### Object Store Snapshot Repository

We assume, an S3-compatible object store is used for taking snapshots, so the necessary plugin `repository-s3` is enabled by default in the Building Block.

Taking snapshots requires a snapshot repository which can be specified in the cluster values file `values-opensearch-cluster.yaml` as

```yaml
cluster:
  general:
    snapshotRepositories:
      - name: "snapshots-s3"
        type: s3
        settings:
          endpoint: objectstorage.fes.cloud.syseleven.net
          bucket: bucket-name
          region: fes
          base_path: os-snapshot
    keystore:
      - secret:
          name: keystore-snapshots-s3-credentials
```

The secret referenced in the `keystore` key must be present in the cluster and can be created with the help of the extension chart's values file `values-opensearch-extension.yaml`

```yaml
s3SnapshotRepositoryCredentials:
  "snapshots-s3":
    accessKey: ACCESS_KEY_ID
    secretKey: SECRET_ACCESS_KEY
```

The naming schema of the resulting secret is `keystore-<name-of-key>-credentials` (here, `keystore-snapshots-s3-credentials`)

### Default Snapshot Policy

It is possible to configure a default snapshot policy through the extension chart, there for add this to its values file `values-opensearch-extension.yaml`

```yaml
snapshots:
  enabled: true
  defaultSnapshotPolicy:
    snapshot_policy_config.json: |
      {
        "description": "Daily snapshot policy",
        "creation": {
          "schedule": {
            "cron": {
              "expression": "0 5 * * *",
              "timezone": "Europe/Berlin"
            }
          },
          "time_limit": "1h"
        },
        "deletion": {
          "schedule": {
            "cron": {
              "expression": "0 1 * * *",
              "timezone": "Europe/Berlin"
            }
          },
          "condition": {
            "max_age": "14d",
            "max_count": 21,
            "min_count": 7
          },
          "time_limit": "1h"
        },
        "snapshot_config": {
          "date_format": "yyyy-MM-dd-HH:mm",
          "timezone": "Europe/Berlin",
          "indices": ["*", "-.opendistro_security"],
          "repository": "snapshots-s3",
          "ignore_unavailable": "true",
          "include_global_state": "false",
          "partial": "true"
        }
      }
```

This would creates a snapshot policy which takes a snapshot daily at `5 am` of all indices without the global state and deletes snapshots older than `14 days`. Here the repository `snapshots-s3` is used which must be present.

The policy shown above is based on this [example](<https://opensearch.org/docs/latest/tuning-your-cluster/availability-and-recovery/snapshots/sm-api/#example>) adjust it to your needs.

### User And Role Management

The official documentation describes how to create users, roles, role bindings, and so on. via custom resources. See [managing security configurations with kubernetes resources](<https://github.com/opensearch-project/opensearch-k8s-operator/blob/main/docs/userguide/main.md#managing-security-configurations-with-kubernetes-resources>)

The user definition references a secret which contains the password. A secret with the name `opensearch-user-passwords` can be created through the extension with the parameter userPasswords. The key is the username of the user that should be created, the value is the user's password. With the definitions in the following example an OpenSearch user, a role and a role binding that connects both is created

```yaml
# values-opensearch-extension.yaml

userPasswords:
  myuser: secure-password
```

```yaml
# values-opensearch-cluster.yaml

roles:
  - name: myindex-crud
    clusterPermissions:
      - "cluster:monitor/state"
      - "cluster_composite_ops"
      - "indices:monitor/settings/get"
    indexPermissions:
      - indexPatterns:
          - myindex*
        allowedActions:
          - "indices:admin/get"
          - "indices:monitor/stats"
          - "create_index"
          - "crud"
          - "manage_aliases"

users:
  - name: myuser
    secretName: opensearch-user-passwords
    secretKey: myuser

usersRoleBinding:
  - name: myuser-myindex-rw
    users:
      - myuser
    roles:
      - myindex-crud
```

## General

### Forward OpenSearch Port 9200 To Localhost Port 9200

```sh
kubectl -n syseleven-opensearch port-forward svc/opensearch-cluster 9200
```

#### API requests using cURL

Obtain basic auth credentials (for example admin)

```sh
# Username
kubectl -n syseleven-opensearch get secret opensearch-cluster-admin-password -o jsonpath='{.data.username}' | base64 -d

# Password
kubectl -n syseleven-opensearch get secret opensearch-cluster-admin-password -o jsonpath='{.data.password}' | base64 -d
```

`HTTP GET to /\_cat/health`

```sh
curl -s -k -uusername:password 'https://localhost:9200/_cat/health?v'
```
