[Helm] Add Logstash Output to Elastic Agent Standalone

changelog/fragments/1760710876-add-support-for-logstash-output-to-elastic-agent-standalone-helm-chart.yaml

@@ -0,0 +1,45 @@
+# REQUIRED
+# Kind can be one of:
+# - breaking-change: a change to previously-documented behavior
+# - deprecation: functionality that is being removed in a later release
+# - bug-fix: fixes a problem in a previous version
+# - enhancement: extends functionality but does not break or fix existing behavior
+# - feature: new functionality
+# - known-issue: problems that we are aware of in a given version
+# - security: impacts on the security of a product or a user’s deployment.
+# - upgrade: important information for someone upgrading from a prior version
+# - other: does not fit into any of the other categories
+kind: feature
+
+# REQUIRED for all kinds
+# Change summary; a 80ish characters long description of the change.
+summary: add support for logstash output to elastic agent standalone helm chart
+
+# REQUIRED for breaking-change, deprecation, known-issue
+# Long description; in case the summary is not enough to describe the change
+# this field accommodate a description without length limits.
+# description:
+
+# REQUIRED for breaking-change, deprecation, known-issue
+# impact:
+
+# REQUIRED for breaking-change, deprecation, known-issue
+# action:
+
+# REQUIRED for all kinds
+# Affected component; usually one of "elastic-agent", "fleet-server", "filebeat", "metricbeat", "auditbeat", "all", etc.
+component:
+
+# AUTOMATED
+# OPTIONAL to manually add other PR URLs
+# PR URL: A link the PR that added the changeset.
+# If not present is automatically filled by the tooling finding the PR where this changelog fragment has been added.
+# NOTE: the tooling supports backports, so it's able to fill the original PR number instead of the backport PR number.
+# Please provide it if you are adding a fragment for a different PR.
+# pr: https://github.com/owner/repo/1234
+
+# AUTOMATED
+# OPTIONAL to manually add other issue URLs
+# Issue URL; optional; the GitHub issue related to this changeset (either closes or is part of).
+# If not present is automatically filled by the tooling with the issue linked to the PR number.
+# issue: https://github.com/owner/repo/1234

deploy/helm/elastic-agent/README.md

@@ -27,28 +27,70 @@ The supported types of outputs are:
 - `ESSecretAuthBasic`: `elasticsearch` output with the connection details specified in a k8s secret
 - `ESSecretAuthAPI`: `elasticsearch` output with the connection details specified in a k8s secret
 - `ESECKRef`: `elasticsearch` output that references by name an Elasticsearch cluster managed by ECK operator
+- `Logstash`: `logstash` output with connection details (hosts, ssl, loadbalance, etc) specified inline in the yaml
 
 | Key | Type | Default | Description |
 |-----|------|---------|-------------|
-| outputs.{name}.type | string | `"ESPlainAuthBasic"` | type of the output [one of `ESPlainAuthBasic`, `ESPlainAuthAPI`, `ESSecretAuthBasic`, `ESSecretAuthAPI`, `ESECKRef`] |
+| outputs.{name}.type | string | `"ESPlainAuthBasic"` | type of the output [one of `ESPlainAuthBasic`, `ESPlainAuthAPI`, `ESSecretAuthBasic`, `ESSecretAuthAPI`, `ESECKRef`, `Logstash`] |
 | outputs.{name}.url | string | `""` | url of the output [required for types `ESPlainAuthBasic` and `ESPlainAuthAPI`] |
 | outputs.{name}.username | string | `""` | the username to use to authenticate with the output [required for type `ESPlainAuthBasic`] |
 | outputs.{name}.password | string | `""` | the password to use to authenticate with the output [required for type `ESPlainAuthBasic`] |
 | outputs.{name}.api_key | string | `""` | the API key use to authenticate with the output [required for type `ESPlainAuthAPI`] |
 | outputs.{name}.secretName | string | `""` | the k8s secret to mount output connection details [required for types `ESSecretAuthBasic` and `ESSecretAuthAPI`] |
 | outputs.{name}.name | string | `""` | name to reference an Elasticsearch cluster managed by ECK [required for type `ESECKRef`] |
-| outputs.{name}.namespace | string | `""` | namespace to reference an Elasticsearch cluster managed by ECK [optional for type `ESECKRef`] |
+
+
+The following fields are specific to Logstash. output type:
+| Key | Type | Default | Example | Description |
+|-----|------|---------|---------|-------------|
+| outputs.{name}.hosts | string array | `""` | `["myhost:5044", "myhost2:5044"]` | An array of Logstash hosts [required] |
+| outputs.{name}.enabled | boolean | `true` | `false` | Enables or disables the output. If set to false, the output is disabled. |
+| outputs.{name}.escape_html | boolean | `false` | `true` | Configures escaping of HTML in strings. Set to true to enable escaping. |
+| outputs.{name}.proxy_url | string | `""` | `"socks5://username:password@socks5host:2233"` |  The URL of the SOCKS5 proxy to use when connecting to the Logstash servers. The value must be a URL with a scheme of `socks5://`. |
+| outputs.{name}.enabled | boolean | `false` | `true` | Determines whether Logstash hostnames are resolved locally when using a proxy. If false and a proxy is used, name resolution occurs on the proxy server. |
+| outputs.{name}.loadbalance | boolean | `false` | `true` | Whether to loadbalance across the logstash hosts |
+| outputs.{name}.ttl | string | `"0"` | `"30s"` | Time to live for a connection to Logstash after which the connection will be reestablished.  |
+| outputs.{name}.slow_start | boolean | `false` | `true` | If true, only a subset of events in a batch of events is transferred per transaction. The number of events to be sent increases up to bulk_max_size if no error is encountered. On error, the number of events per transaction is reduced again.  |
+| outputs.{name}.pipelining | integer | 2 | `5` | The number of batches to send asynchronously to Logstash while waiting for an ACK from Logstash. The output becomes blocking after the specified number of batches are written. Specify 0 to turn off pipelining. |
+
+For `Logstash` extra fields can be specified inline the yaml following these guidelines:
+ - ["Performance tuning settings"](https://www.elastic.co/docs/reference/fleet/logstash-output#output-logstash-performance-tuning-settings)
+ - ["Memory queue settings"](https://www.elastic.co/docs/reference/fleet/logstash-output#output-logstash-memory-queue-settings)
+
+ For all outputs(except `ESECKRef`) these extra fields can be specified inline the yaml following these guidelines
+ - ["SSL/TLS Settings"](https://www.elastic.co/docs/reference/fleet/elastic-agent-ssl-configuration)
+  *NOTE* ssl.certificate_authorities is defined as ssl.certificateAuthorities
+  *NOTE* ssl.verification_mode is defined as ssl.verificationMode
+
+The following variables can be specified using valueFrom syntax:
+ * ssl.certificate
+ * ssl.key
+ * ssl.key_passphrase
+ * ssl.certificateAuthorities
+
+ The valueFrom syntax allows you to set a variable two different ways:
+```
+variable_name:
+    valueFromSecret:
+        name: <secret name>
+        key: <secret key>
+```
+
+or
+
+```
+variable_name:
+    value: "text"
+```
+
 Examples of Helm chart arguments to define an output with name `myOutput`:
 - `ESPlainAuthBasic`: `--set outputs.myOutput.url=https://elasticsearch:9200 --set outputs.myOutput.username=changeme --set outputs.myOutput.password=changeme`
 - `ESPlainAuthAPI`: `--set outputs.myOutput.url=https://elasticsearch:9200 --set outputs.myOutput.api_key=token`
 - `ESSecretAuthBasic`: `--set outputs.myOutput.type=ESSecretAuthBasic --set outputs.myOutput.secretName=k8s_secret_name` (required keys in the k8s secret are `url`, `username`, `password`)
 - `ESSecretAuthAPI`: `--set outputs.myOutput.type=ESSecretAuthAPI --set outputs.myOutput.secretName=k8s_secret_name` (required keys in the k8s secret are `url`, `api_key`)
 - `ESECKRef`: `--set outputs.myOutput.type=ESECKRef --set outputs.myOutput.name=eck_es_cluster_name`
+- `Logstash`: `--set outputs.myOutput.type=Logstash --set outputs.myOutput.hosts=["myHost:5044"] --set outputs.myOutput.ssl.enabled=true --set-file outputs.myOuput.ssl.certificateAuthorities[0].value=/path/to/ca.crt`
 
-For `ESPlainAuthBasic`, `ESPlainAuthAPI` `ESSecretAuthBasic`, `ESSecretAuthAPI` extra fields can be specified inline the yaml following these guidelines (`ESECKRef` doesn't support them):
- - ["Data parsing, filtering, and manipulation settings"](`https://www.elastic.co/guide/en/fleet/current/elasticsearch-output.html#output-elasticsearch-data-parsing-settings`)
- - ["Performance tuning settings"](https://www.elastic.co/guide/en/fleet/current/elasticsearch-output.html#output-elasticsearch-performance-tuning-settings)
- - ["Memory queue settings"](https://www.elastic.co/guide/en/fleet/current/elasticsearch-output.html#output-elasticsearch-memory-queue-settings)
 
 ### 2 - Kubernetes integration
 

deploy/helm/elastic-agent/examples/kubernetes-logstash-output/README.md

@@ -0,0 +1,96 @@
+# Example Kubernetes Integration with Logstash output
+
+In this example we install the built-in `kubernetes` integration with the default built-in values and a different agent output with ssl settings that allow to connect to a Logstash cluster that is exposed with mtls using a self-signed certificate.
+
+## Prerequisites:
+1. Build the dependencies of the Helm chart
+    ```console
+    helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
+    helm dependency build ../../
+    ```
+2. A k8s secret that contains the client cert and key for [mtls to a logstash cluster](https://www.elastic.co/docs/reference/fleet/secure-logstash-connections) stored in the files tls.crt and tls.key.
+    ```console
+    kubectl create secret tls cert-secret \
+       --cert=tls.crt \
+       --key=tls.key
+    ```
+
+3. `kubernetes` integration assets installed through Kibana ([Kibana - Install and uninstall Elastic Agent integration assets](https://www.elastic.co/guide/en/fleet/current/install-uninstall-integration-assets.html))
+4. The CA certificate, that validates the self-signed server certificate, stored in the file `ca.crt`. Example content of the `ca.crt` file.
+     ```
+     -----BEGIN CERTIFICATE-----
+     MIIDSjCCAjKgAwIBAgIRALfMeXFmYLUW4HaNXLzfP4cwDQYJKoZIhvcNAQELBQAw
+     LzETMBEGA1UECxMKbW9uaXRvcmluZzEYMBYGA1UEAxMPbW9uaXRvcmluZy1odHRw
+     MB4XDTI0MTIxMTEwMTMzNVoXDTI1MTIxMTEwMjMzNVowLzETMBEGA1UECxMKbW9u
+     aXRvcmluZzEYMBYGA1UEAxMPbW9uaXRvcmluZy1odHRwMIIBIjANBgkqhkiG9w0B
+     AQEFAAOCAQ8AMIIBCgKCAQEAsljXOJrCsvZGHr2SroKUGJOnJwtz8VTx2spQ96OO
+     8Q+Tw8gX5C32bjplwAeQsnZ7i5YRRLneaG6NXJuaUEDefsKeG6jdN/bjce+Sz5xm
+     U6guXe3TuIyk0+UoFtOzZ1lYUNk6lg9+60iOllRO3xI7SwxqKAaC4KKs7QL1jQCR
+     Q14QedcPrS4v76OT+TJvYWrbTFLtYYvfJDGop5EE90v7iB5j0ehSLjfC2R4CD5Kr
+     OSYJrGqnhnznbUUjulVqCkPKmgZdcvcIBn4NnZlN6oYzwhRHSSj6r3sy11j3A6SA
+     7KeG+IlY+LmRtrj85tiRJ3pXz1FD2d/Mf6cNI6lBGRrNZwIDAQABo2EwXzAOBgNV
+     HQ8BAf8EBAMCAoQwHQYDVR0lBBYwFAYIKwYBBQUHAwEGCCsGAQUFBwMCMA8GA1Ud
+     EwEB/wQFMAMBAf8wHQYDVR0OBBYEFMgVU7RwXciOOz18FcQDTQZXy9gIMA0GCSqG
+     SIb3DQEBCwUAA4IBAQCgOSe2s3Xc0QKR+86xmoAADpoe7SFT0Yyh3rMjL+0p02m3
+     CqrILqCRNFu9az8gc47hUt9Crb1BXmTR0Sb23M1NvGmR2D2K7CLp/SvkAP6RlB4M
+     dZ70UKw4ohq+VSSSiLOoHYdlH46xtunLL31GLYRwD+OgeKAc5pwqWgZkndzxrouB
+     uNyoxB5NGvaVUqIouILQ9V2fvraCNf+RxuQ0AaPxdt/CNpFaXpbJBuXJCphlydu0
+     KztVqRv5EZjuYpcXDfGP9BEvMy6o895H4iG0M2wb2e3WEDo6jH5pecZfc4yz8iae
+     jLwbOPbWqOGRkxTMLOV6Q1dtr09zf2SuOQuxm7F2
+     -----END CERTIFICATE-----
+     ```
+5. A logstash instance running with the server tls certificates that match the client set used by elastic agent. Running a pipeline similar to the below:
+
+```
+    input {
+      elastic_agent {
+        port => 5044
+        ssl_enabled => true
+        ssl_certificate_authorities => ["/opt/logstash/ssl/ca.crt"]
+        ssl_certificate => "/opt/logstash/ssl/logstash.crt"
+        ssl_key => "/opt/logstash/ssl/logstash.pkcs8.key"
+        ssl_client_authentication => "required"
+      }
+    }
+    filter {
+       elastic_integration {
+         cloud_id => "${CLOUD_ID}"
+         username => "${ES_USER}"
+         password => "${ES_PASSWORD}"
+       }
+    }
+    output {
+        elasticsearch {
+            cloud_id => "${CLOUD_ID}"
+            user => "${ES_USER}"
+            password => "${ES_PASSWORD}"
+            data_stream => "true"
+            manage_template => "false"
+        }
+    }
+```
+## Run:
+```console
+helm install elastic-agent ../../ \
+     -f ./agent-kubernetes-values.yaml \
+     --set outputs.test.hosts[0]="logstash-agent.default.svc.cluster.local:5044" \
+     --set outputs.test.type=Logstash \
+     --set-file outputs.test.ssl.certificateAuthorities[0].value=ca.crt \
+     --set outputs.test.ssl.certificate.valueFromSecret.key=tls.crt \
+     --set outputs.test.ssl.certificate.valueFromSecret.name=cert-secret \
+     --set outputs.test.ssl.key.valueFromSecret.key=tls.key \
+     --set outputs.test.ssl.key.valueFromSecret.name=cert-secret \
+     --set agent.presets.perNode.agent.monitoring.use_output=test \
+     --set agent.presets.clusterWide.agent.monitoring.use_output=test \
+     --set kubernetes.output=test
+```
+
+## Validate:
+
+1. `kube-state metrics` is installed with this command `kubectl get deployments -n kube-system kube-state-metrics`.
+2. Kubernetes data ship to the Logstash Cluster of the `test` output.
+3. The Kibana `kubernetes`-related dashboards should start showing up the respective info.
+
+## Note:
+
+1. If you want to disable kube-state-metrics installation with the elastic-agent Helm chart, you can set `kube-state-metrics.enabled=false` in the Helm chart. The helm chart will use the value of `kubernetes.state.host` to configure the elastic-agent input.

deploy/helm/elastic-agent/examples/kubernetes-logstash-output/agent-kubernetes-values.yaml

@@ -0,0 +1,77 @@
+kubernetes:
+  enabled: true
+  output: test
+  namespace: test
+  scheduler:
+    enabled: true
+  controller_manager:
+    enabled: true
+
+outputs:
+  test:
+    type: Logstash
+    hosts:
+    - myhost:5443
+    loadbalance: true
+    ttl: "10s"
+    slow_start: false
+    timeout: "30s"
+
+    workers: 5
+    pipelining: 1
+    bulk_max_size: 1600
+    queue:
+      mem: 
+        flush:
+         timeout: "30s"
+         min_events: 1600
+        events:  3200
+    max_retries: 3
+    compression_level: 5
+    backoff:
+      max: "30s"
+      init: "1s"
+    allow_older_versions: false
+    ssl:
+      certificate: 
+        valueFromSecret: 
+            name: cert-secret
+            key: cert
+      key: 
+        valueFromSecret:
+            name: cert-secret
+            key: key
+      verificationMode: certificate
+      certificateAuthorities:
+        - value: |-
+            -----BEGIN CERTIFICATE-----
+            MIIDSjCCAjKgAwIBAgIRALx9cV4kYFWe1ZtMXLzfX4cwDQYJKoZIhvcNAQELBQAw
+            LzETMBEGA1UECxMKbW9uaXRvcmluZzEYMBYGA1UEAxMPbW9uaXRvcmluZy1odHRw
+            MB4XDTI1MDQwODEyMDAwMFoXDTI2MDQwODEyMDAwMFowLzETMBEGA1UECxMKbW9u
+            aXRvcmluZzEYMBYGA1UEAxMPbW9uaXRvcmluZy1odHRwMIIBIjANBgkqhkiG9w0B
+            AQEFAAOCAQ8AMIIBCgKCAQEAwk7mDcV4hO6WKnz2z2gC+ODb2xF6aP6sB5o3rZ2F
+            R0vbPY9EYI56bxSB3zT5mx0I3XsdKy8JK3AeY32K5CPv7tkHeGqTxJfYKwKXWpO4
+            rN9i3PtZZ3ZX5iPBiGUKuZJH2HQolZt/8OoyH5n3FEs/zNVbrXL6RuMjVzRGyJhR
+            UeYpXib35SKvwWWNHh9fHV1m8WUdJxvEY9ohkry0ENj6tzB9Om+yKq5Scdw3hC5h
+            X1f6+fY7ElZjVQ2lXnyrRHKpLjcDhI9cRm7r1qeKcHQMxz4Rzk3NYOBfMiEn+pns
+            OLqUz9tK0q3VgbIUlFe94CEZ2DDkRpsIqyfRTUutOwIDAQABo2EwXzAOBgNVHQ8B
+            Af8EBAMCAoQwHQYDVR0lBBYwFAYIKwYBBQUHAwEGCCsGAQUFBwMCMA8GA1UdEwEB
+            /wQFMAMBAf8wHQYDVR0OBBYEFDUYT4RsXgjJJz38XfQHTTZVr8gOMA0GCSqGSIb3
+            DQEBCwUAA4IBAQCvIXs1b8Pe4W5PbqQtnAV+wsyq4XqB7kFNTfZBx8eh5RMqG4y8
+            pTRSDM3I+hMI1K2XpMyUM4dyaZPz2yxOZ7UgvxFwLdHDb9uYaXJ38Rm3FxV9Z+Lr
+            ZPbXce2po5WT1em8MyxLgTStbMaXYHlsk09Fj4hF9Gy93bz4iVR7cUbDgkZd7ZPP
+            uN2bDD9kU+1iyqYv44YGbEv0UTKhrT37GkBzZ3H4cGy8sKnuwz7Z0I8v+FZzKN4L
+            Knk2MCWxZQ6t4nnqmkFoK4nSlAmW5Fnj7c2YwPYsZnscEcFlS0YDNmwC8wvDdDHL
+            tD0YrMZxZDh0LAGWAW72rT+mSPCWq5eB2ZtS
+            -----END CERTIFICATE-----
+agent:
+  unprivileged: true
+  presets:
+    perNode:
+      agent:
+        monitoring:
+          use_output: test
+    clusterWide:
+      agent:
+        monitoring:
+          use_output: test