diff --git a/.github/workflows/pr-example-validation.yaml b/.github/workflows/pr-example-validation.yaml new file mode 100644 index 000000000..9d371f6db --- /dev/null +++ b/.github/workflows/pr-example-validation.yaml @@ -0,0 +1,69 @@ +name: Validate example configurations in PRs + +on: + pull_request: + paths: + - '**/*.md' + +permissions: {} + +# Avoid multiple commits on the same PR racing to update the comment. +concurrency: + group: ${{ github.workflow }}-${{ github.event.pull_request.number }} + cancel-in-progress: true + +jobs: + validate-configs: + name: Validate example configurations in changed Markdown files + runs-on: ubuntu-latest + permissions: + contents: read + steps: + - name: Check out code + uses: actions/checkout@v7 + with: + fetch-depth: 0 + persist-credentials: false + + - name: Get changed Markdown files + # We do not use the usual changed files action because of previous security issues. + id: changed_files + # This complex command finds files changed in the PR, filtering for `.md` and also ensuring they are in the `docs` directory. + # It ensures we only process files that are part of the PR and match our pattern. + # Note: The `github.event.pull_request.base.sha` is used to get the base commit of the PR, which is important for accurate diffing. + # + # On pull requests, HEAD refers to the merge commit. + # We want to compare against the base of the PR. + # Use git diff-tree to get files changed between the PR base and the current commit. + # --name-only lists only the names of the files. + # --diff-filter=AMCR will only show Added, Modified, Copied, Renamed files. + # We grep for *.md files you are interested in. + # The 'docs/' prefix is a common convention but adjust as necessary. + run: | + changed_md=$(git diff-tree --no-commit-id --name-only --diff-filter=AMCR ${{ github.event.pull_request.base.sha }}...HEAD | grep './.*\.md$' || true) + echo "Changed Markdown files: $changed_md" + echo "list=${changed_md}" >> $GITHUB_OUTPUT + shell: bash + + - name: Validate example configurations provided in any changed files + run: | + if [ -z "${CHANGED_MD_FILES:-}" ]; then + echo "No Markdown files changed in this PR. Skipping validation." + exit 0 + fi + + error_count=0 + + # Loop through each changed file + for FILE in $CHANGED_MD_FILES; do + echo "Processing changed file: $FILE" + ./scripts/test-config.sh "$FILE" || error_count=$((error_count + 1)) + done + + if [ $error_count -ne 0 ]; then + echo "ERROR: Validation failed for $error_count file(s)." + exit 1 + fi + shell: bash + env: + CHANGED_MD_FILES: ${{ steps.changed_files.outputs.list }} diff --git a/installation/downloads/docker.md b/installation/downloads/docker.md index 758cbd3c8..312e5ebe4 100644 --- a/installation/downloads/docker.md +++ b/installation/downloads/docker.md @@ -15,7 +15,7 @@ docker run -ti cr.fluentbit.io/fluent/fluent-bit Use the following command to start Fluent Bit while using a configuration file: {% tabs %} -{% tab title="fluent-bit.conf" %} +{% tab title="Legacy Configuration: fluent-bit.conf" %} ```shell docker run -ti -v ./fluent-bit.conf:/fluent-bit/etc/fluent-bit.conf \ @@ -24,7 +24,7 @@ docker run -ti -v ./fluent-bit.conf:/fluent-bit/etc/fluent-bit.conf \ {% endtab %} -{% tab title="fluent-bit.yaml" %} +{% tab title="YAML configuration: fluent-bit.yaml" %} ```shell docker run -ti -v ./fluent-bit.yaml:/fluent-bit/etc/fluent-bit.yaml \ diff --git a/pipeline/buffering.md b/pipeline/buffering.md index bb055ecec..f1947b465 100644 --- a/pipeline/buffering.md +++ b/pipeline/buffering.md @@ -102,7 +102,8 @@ pipeline: # Inherits storage.type: filesystem from service - name: mem - storage.type: memory # Overrides the inherited default + # Overrides the inherited default + storage.type: memory ``` {% endtab %} @@ -124,7 +125,8 @@ pipeline: [INPUT] Name mem - Storage.Type memory # Overrides the inherited default + # Overrides the inherited default + Storage.Type memory ``` {% endtab %} diff --git a/pipeline/filters/geoip2-filter.md b/pipeline/filters/geoip2-filter.md index bbd981580..5e4f969c0 100644 --- a/pipeline/filters/geoip2-filter.md +++ b/pipeline/filters/geoip2-filter.md @@ -33,7 +33,7 @@ The following configuration processes the incoming `remote_addr` and appends cou pipeline: inputs: - name: dummy - dummy: {"remote_addr": "8.8.8.8"} + dummy: '{"remote_addr": "8.8.8.8"}' filters: - name: geoip2 diff --git a/pipeline/filters/kubernetes.md b/pipeline/filters/kubernetes.md index 028952caf..5501e90fc 100644 --- a/pipeline/filters/kubernetes.md +++ b/pipeline/filters/kubernetes.md @@ -360,10 +360,11 @@ pipeline: {% tab title="fluent-bit.conf" %} ```text -[PARSER] - Name custom-tag - Format regex - Regex ^(?[^_]+)\.(?[a-z0-9](?:[-a-z0-9]*[a-z0-9])?(?:\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*)\.(?.+)\.(?[a-z0-9]{64}) +# Ensure this is added to parsers_file loaded in the SERVICE section +# [PARSER] +# Name custom-tag +# Format regex +# Regex ^(?[^_]+)\.(?[a-z0-9](?:[-a-z0-9]*[a-z0-9])?(?:\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*)\.(?.+)\.(?[a-z0-9]{64}) [INPUT] Name tail @@ -488,7 +489,7 @@ pipeline: parser: docker docker_mode: on mem_buf_limit: 50MB - skip_login_lines: on + skip_long_lines: on refresh_interval: 10 filters: diff --git a/pipeline/filters/parser.md b/pipeline/filters/parser.md index 1db516612..60e9dbfbc 100644 --- a/pipeline/filters/parser.md +++ b/pipeline/filters/parser.md @@ -27,7 +27,7 @@ The plugin needs a parser file which defines how to parse each field. This is an example of parsing a record `{"data":"100 0.5 true This is example"}`. {% tabs %} -{% tab title="fluent-bit.yaml" %} +{% tab title="parser.yaml" %} ```yaml parsers: @@ -37,7 +37,7 @@ parsers: ``` {% endtab %} -{% tab title="fluent-bit.conf" %} +{% tab title="parser.conf" %} ```text [PARSER] diff --git a/pipeline/inputs/blob.md b/pipeline/inputs/blob.md index aa393a36f..d02704ca2 100644 --- a/pipeline/inputs/blob.md +++ b/pipeline/inputs/blob.md @@ -177,8 +177,8 @@ This example excludes certain file patterns and uses filesystem storage for bett pipeline: inputs: - name: blob - path: /data/artifacts/**/* - exclude_pattern: *.tmp,*.bak,*.old + path: '/data/artifacts/**/*' + exclude_pattern: '*.tmp,*.bak,*.old' storage.type: filesystem storage.pause_on_chunks_overlimit: true mem_buf_limit: 50M diff --git a/pipeline/inputs/cpu-metrics.md b/pipeline/inputs/cpu-metrics.md index 7546e7f5b..669ab7702 100644 --- a/pipeline/inputs/cpu-metrics.md +++ b/pipeline/inputs/cpu-metrics.md @@ -82,7 +82,7 @@ pipeline: {% endtab %} {% tab title="fluent-bit.conf" %} -```shell +```text [INPUT] Name cpu Tag my_cpu diff --git a/pipeline/inputs/kafka.md b/pipeline/inputs/kafka.md index 9b460ec5f..404d06e9c 100644 --- a/pipeline/inputs/kafka.md +++ b/pipeline/inputs/kafka.md @@ -95,8 +95,16 @@ pipeline: filters: - name: lua match: '*' - script: kafka.lua call: modify_kafka_message + code: | + local count = 0 + function modify_kafka_message(tag, timestamp, record) + count = count + 1 + local payload = record.payload + payload.topic = record.topic + payload.status = 'processed by fluent-bit, total records: '..tostring(count) + return 1, timestamp, payload + end outputs: - name: kafka @@ -118,8 +126,8 @@ pipeline: [FILTER] Name lua Match * - Script kafka.lua Call modify_kafka_message + Code local count = 0 function modify_kafka_message(tag, timestamp, record) count = count + 1 local payload = record.payload payload.topic = record.topic payload.status = 'processed by fluent-bit, total records: '..tostring(count) return 1, timestamp, payload end [OUTPUT] Name kafka @@ -134,7 +142,18 @@ The previous example will connect to the broker listening on `kafka-broker:9092` Since the payload will be in JSON format, the plugin is configured to parse the payload with `format json`. -Every message received is then processed with `kafka.lua` and sent back to the `fb-sink` topic of the same broker. +Every message received is then processed with `kafka.lua` and sent back to the `fb-sink` topic of the same broker, this is shown inline for each configuration but is as follows: + +```lua +local count = 0 +function modify_kafka_message(tag, timestamp, record) + count = count + 1 + local payload = record.payload + payload.topic = record.topic + payload.status = 'processed by fluent-bit, total records: '..tostring(count) + return 1, timestamp, payload +end +``` The example can be executed locally with `make start` in the `examples/kafka_filter` directory (`docker/compose` is used). diff --git a/pipeline/inputs/tail.md b/pipeline/inputs/tail.md index 746b1b5eb..2e0a61e9f 100644 --- a/pipeline/inputs/tail.md +++ b/pipeline/inputs/tail.md @@ -272,8 +272,8 @@ pipeline: path: /var/log/syslog outputs: - - stdout: - match: * + - name: stdout + match: '*' ``` {% endtab %} @@ -368,13 +368,13 @@ pipeline: {% tab title="fluent-bit.conf" %} ```text -# Note this is generally added to parsers.conf and referenced in [SERVICE] -[PARSER] - Name multiline - Format regex - Regex /(?