From 125740b1c4ec070d147c069001777dd60446b23c Mon Sep 17 00:00:00 2001 From: Lynette Miles Date: Tue, 1 Jul 2025 12:57:49 -0700 Subject: [PATCH 1/3] Pipeline: input: stdin: style Signed-off-by: Lynette Miles --- pipeline/inputs/standard-input.md | 168 +++++++++++++++++------------- 1 file changed, 96 insertions(+), 72 deletions(-) diff --git a/pipeline/inputs/standard-input.md b/pipeline/inputs/standard-input.md index d65fb1c9d..8e7e975fc 100644 --- a/pipeline/inputs/standard-input.md +++ b/pipeline/inputs/standard-input.md @@ -1,58 +1,75 @@ -# Standard Input +# Standard input -The **stdin** plugin supports retrieving a message stream from the standard input interface \(stdin\) of the Fluent Bit process. -In order to use it, specify the plugin name as the input, e.g: +The _Standard input_ plugin supports retrieving a message stream from the standard input interface (`stdin`) of the Fluent Bit process. +To use it, specify the plugin name as the input. For example: ```bash -$ fluent-bit -i stdin -o stdout + fluent-bit -i stdin -o stdout ``` -If the stdin stream is closed (end-of-file), the stdin plugin will instruct -Fluent Bit to exit with success (0) after flushing any pending output. +If the `stdin` stream is closed (`end-of-file`), the plugin instruct Fluent Bit to exit with success (`0`) after flushing any pending output. + +## Configuration parameters + +The plugin supports the following configuration parameters: + +| Key | Description | Default | +| :--- | :--- | :--- | +| `Buffer_Size` | Set the buffer size to read data. This value is used to increase buffer size and must be set according to the [Unit Size](../../administration/configuring-fluent-bit/unit-sizes.md) specification. | `16k` | +| `Parser` | The name of the parser to invoke instead of the default JSON input parser. | _none_ | +| `Threaded` | Indicates whether to run this input in its own [thread](../../administration/multithreading.md#inputs). | `false` | ## Input formats -If no parser is configured for the stdin plugin, it expects *valid JSON* input data in one of the following formats: +If no parser is configured for the `stdin` plugin, it expects valid JSON input data in one of the following formats: -1. A JSON object with one or more key-value pairs: `{ "key": "value", "key2": "value2" }` -3. A 2-element JSON array in [Fluent Bit Event](../../concepts/key-concepts.md#event-or-record) format, which may be: - * `[TIMESTAMP, { "key": "value" }]` where TIMESTAMP is a floating point value representing a timestamp in seconds; or - * from Fluent Bit v2.1.0, `[[TIMESTAMP, METADATA], { "key": "value" }]` where TIMESTAMP has the same meaning as above and METADATA is a JSON object. +- A JSON object with one or more key-value pairs: `{ "key": "value", "key2": "value2" }` +- A 2-element JSON array in [Fluent Bit Event](../../concepts/key-concepts.md#event-or-record) format, which can be: + - `[TIMESTAMP, { "key": "value" }]` where TIMESTAMP is a floating point value representing a timestamp in seconds. + - From Fluent Bit v2.1.0, `[[TIMESTAMP, METADATA], { "key": "value" }]` where _`TIMESTAMP`_ has the same meaning as previous and _`METADATA`_ is a JSON object. Multi-line input JSON is supported. -Any input data that is *not* in one of the above formats will cause the plugin to log errors like: +Any input data which isn't in one of the supported formats will cause the plugin to log errors like: -``` +```text [debug] [input:stdin:stdin.0] invalid JSON message, skipping [error] [input:stdin:stdin.0] invalid record found, it's not a JSON map or array ``` -To handle inputs in other formats, a parser must be explicitly specified in the configuration for the `stdin` plugin. See [parser input example](#parser-input-example) for sample configuration. +To handle inputs in other formats, a parser must be explicitly specified in the configuration for the `stdin` plugin. See [parser input example](#parser-input) for sample configuration. ## Log event timestamps -The Fluent Bit event timestamp will be set from the input record if the 2-element event input is used or a custom parser configuration supplies a timestamp. Otherwise the event timestamp will be set to the timestamp at which the record is read by the stdin plugin. +The Fluent Bit event timestamp will be set from the input record if the two-element event input is used or a custom parser configuration supplies a timestamp. Otherwise the event timestamp will be set to the timestamp at which the record is read by the `stdin` plugin. ## Examples -### Json input example +### JSON input -A better example to demonstrate how it works will be through a _Bash_ script that generates messages and writes them to [Fluent Bit](http://fluentbit.io). Write the following content in a file named _test.sh_: +To demonstrate how the plugin works, you can use a `bash` script that generates +messages and writes them to [Fluent Bit](http://fluentbit.io). -```bash -#!/bin/sh +1. Write the following content in a file named `test.sh`: -for ((i=0; i<=5; i++)); do - echo -n "{\"key\": \"some value\"}" - sleep 1 -done -``` + ```bash + #!/bin/sh -Now lets start the script and [Fluent Bit](http://fluentbit.io): + for ((i=0; i<=5; i++)); do + echo -n "{\"key\": \"some value\"}" + sleep 1 + done + ``` -```bash -$ bash test.sh | fluent-bit -q -i stdin -o stdout +1. Start the script and [Fluent Bit](http://fluentbit.io): + + ```bash + bash test.sh | fluent-bit -q -i stdin -o stdout + ``` + +The command should return output like the following: + +```text [0] stdin.0: [[1684196745.942883835, {}], {"key"=>"some value"}] [0] stdin.0: [[1684196746.938949056, {}], {"key"=>"some value"}] [0] stdin.0: [[1684196747.940162493, {}], {"key"=>"some value"}] @@ -61,9 +78,9 @@ $ bash test.sh | fluent-bit -q -i stdin -o stdout [0] stdin.0: [[1684196750.943721442, {}], {"key"=>"some value"}] ``` -### Json input with timestamp example +### JSON input with timestamp -An input event timestamp may also be supplied. Replace `test.sh` with: +An input event timestamp can also be supplied. Replace `test.sh` with: ```bash #!/bin/sh @@ -81,10 +98,15 @@ for ((i=0; i<=5; i++)); do done ``` -Re-run the sample command. Note that the timestamps output by Fluent Bit are now one day old because Fluent Bit used the input message timestamp. +Re-run the sample command. Timestamps output by Fluent Bit are now one day old because Fluent Bit used the input message timestamp. ```bash -$ bash test.sh | fluent-bit -q -i stdin -o stdout +bash test.sh | fluent-bit -q -i stdin -o stdout +``` + +Which returns the following: + +```text [0] stdin.0: [[1684110480.028171300, {}], {"realtimestamp"=>1684196880.030070}] [0] stdin.0: [[1684110481.033753395, {}], {"realtimestamp"=>1684196881.034741}] [0] stdin.0: [[1684110482.036730051, {}], {"realtimestamp"=>1684196882.037704}] @@ -93,10 +115,10 @@ $ bash test.sh | fluent-bit -q -i stdin -o stdout [0] stdin.0: [[1684110485.048710107, {}], {"realtimestamp"=>1684196885.049651}] ``` -### Json input with metadata example +### JSON input with metadata -Additional metadata is also supported on Fluent Bit v2.1.0 and above by replacing the timestamp -with a 2-element object, e.g.: +Additional metadata is supported in Fluent Bit v2.1.0 and later by replacing the timestamp +with a two-element object. For example: ```bash #!/bin/sh @@ -116,8 +138,15 @@ for ((i=0; i<=5; i++)); do done ``` +Run test using the command: + +```bash +bash ./test.sh | fluent-bit -q -i stdin -o stdout ``` -$ bash ./test.sh | fluent-bit -q -i stdin -o stdout + +Which returns results like the following: + +```text [0] stdin.0: [[1684110513.060139417, {"metakey"=>"metavalue"}], {"realtimestamp"=>1684196913.061017}] [0] stdin.0: [[1684110514.063085317, {"metakey"=>"metavalue"}], {"realtimestamp"=>1684196914.064145}] [0] stdin.0: [[1684110515.066210508, {"metakey"=>"metavalue"}], {"realtimestamp"=>1684196915.067155}] @@ -126,23 +155,19 @@ $ bash ./test.sh | fluent-bit -q -i stdin -o stdout [0] stdin.0: [[1684110518.075428724, {"metakey"=>"metavalue"}], {"realtimestamp"=>1684196918.076292}] ``` -On older Fluent Bit versions records in this format will be discarded. Fluent Bit will log: +On older Fluent Bit versions records in this format will be discarded. If the log level permits, Fluent Bit will log: -``` +```text [ warn] unknown time format 6 ``` -if the log level permits. - -### Parser input example +### Parser input -To capture inputs in other formats, specify a parser configuration for the -`stdin` plugin. +To capture inputs in other formats, specify a parser configuration for the `stdin` plugin. -For example, if you want to read raw messages line-by-line and forward them you -could use a `parser.conf` that captures the whole message line: +For example, if you want to read raw messages line byline and forward them you could use a `parser.conf` that captures the whole message line: -``` +```text [PARSER] name stringify_message format regex @@ -150,23 +175,12 @@ could use a `parser.conf` that captures the whole message line: regex ^(?.*) ``` -then use that in the `parser` clause of the stdin plugin in the `fluent-bit.conf`: +YOu can then use that in the `parser` clause of the `stdin` plugin in the `fluent-bit.conf`: {% tabs %} -{% tab title="fluent-bit.conf" %} -``` -[INPUT] - Name stdin - Tag stdin - Parser stringify_message - -[OUTPUT] - Name stdout - Match * -``` -{% endtab %} {% tab title="fluent-bit.yaml" %} + ```yaml pipeline: inputs: @@ -177,14 +191,35 @@ pipeline: - name: stdout match: '*' ``` + +{% endtab %} + +{% tab title="fluent-bit.conf" %} + +```text +[INPUT] + Name stdin + Tag stdin + Parser stringify_message + +[OUTPUT] + Name stdout + Match * +``` + {% endtab %} {% endtabs %} Fluent Bit will now read each line and emit a single message for each input -line: +line, using the following command: +```shell +seq 1 5 | /opt/fluent-bit/bin/fluent-bit -c fluent-bit.conf -R parser.conf -q ``` -$ seq 1 5 | /opt/fluent-bit/bin/fluent-bit -c fluent-bit.conf -R parser.conf -q + +Which returns output similar to: + +```text [0] stdin: [1681358780.517029169, {"message"=>"1"}] [1] stdin: [1681358780.517068334, {"message"=>"2"}] [2] stdin: [1681358780.517072116, {"message"=>"3"}] @@ -193,15 +228,4 @@ $ seq 1 5 | /opt/fluent-bit/bin/fluent-bit -c fluent-bit.conf -R parser.conf -q $ ``` -In real-world deployments it is best to use a more realistic parser that splits -messages into real fields and adds appropriate tags. - -## Configuration Parameters - -The plugin supports the following configuration parameters: - -| Key | Description | Default | -| :--- | :--- | :--- | -| Buffer\_Size | Set the buffer size to read data. This value is used to increase buffer size. The value must be according to the [Unit Size](../../administration/configuring-fluent-bit/unit-sizes.md) specification. | 16k | -| Parser | The name of the parser to invoke instead of the default JSON input parser | | -| Threaded | Indicates whether to run this input in its own [thread](../../administration/multithreading.md#inputs). | `false` | +In production deployments it's best to use a parser that splits messages into real fields and adds appropriate tags. From 1d526614a0cd11d5089e0b4f52355ae4218c93cd Mon Sep 17 00:00:00 2001 From: Lynette Miles <6818907+esmerel@users.noreply.github.com> Date: Tue, 1 Jul 2025 13:43:51 -0700 Subject: [PATCH 2/3] Apply suggestions from code review Co-authored-by: Craig Norris <112565517+cnorris-cs@users.noreply.github.com> Signed-off-by: Lynette Miles <6818907+esmerel@users.noreply.github.com> --- pipeline/inputs/standard-input.md | 13 +++++-------- 1 file changed, 5 insertions(+), 8 deletions(-) diff --git a/pipeline/inputs/standard-input.md b/pipeline/inputs/standard-input.md index 8e7e975fc..11baa1bd7 100644 --- a/pipeline/inputs/standard-input.md +++ b/pipeline/inputs/standard-input.md @@ -7,7 +7,7 @@ To use it, specify the plugin name as the input. For example: fluent-bit -i stdin -o stdout ``` -If the `stdin` stream is closed (`end-of-file`), the plugin instruct Fluent Bit to exit with success (`0`) after flushing any pending output. +If the `stdin` stream is closed (`end-of-file`), the plugin instructs Fluent Bit to exit with success (`0`) after flushing any pending output. ## Configuration parameters @@ -47,8 +47,7 @@ The Fluent Bit event timestamp will be set from the input record if the two-elem ### JSON input -To demonstrate how the plugin works, you can use a `bash` script that generates -messages and writes them to [Fluent Bit](http://fluentbit.io). +To demonstrate how the plugin works, you can use a `bash` script that generates messages and writes them to [Fluent Bit](http://fluentbit.io). 1. Write the following content in a file named `test.sh`: @@ -117,8 +116,7 @@ Which returns the following: ### JSON input with metadata -Additional metadata is supported in Fluent Bit v2.1.0 and later by replacing the timestamp -with a two-element object. For example: +Additional metadata is supported in Fluent Bit v2.1.0 and later by replacing the timestamp with a two-element object. For example: ```bash #!/bin/sh @@ -165,7 +163,7 @@ On older Fluent Bit versions records in this format will be discarded. If the lo To capture inputs in other formats, specify a parser configuration for the `stdin` plugin. -For example, if you want to read raw messages line byline and forward them you could use a `parser.conf` that captures the whole message line: +For example, if you want to read raw messages line by line and forward them, you could use a `parser.conf` that captures the whole message line: ```text [PARSER] @@ -175,7 +173,7 @@ For example, if you want to read raw messages line byline and forward them you c regex ^(?.*) ``` -YOu can then use that in the `parser` clause of the `stdin` plugin in the `fluent-bit.conf`: +You can then use that in the `parser` clause of the `stdin` plugin in the `fluent-bit.conf` file: {% tabs %} @@ -211,7 +209,6 @@ pipeline: {% endtabs %} Fluent Bit will now read each line and emit a single message for each input -line, using the following command: ```shell seq 1 5 | /opt/fluent-bit/bin/fluent-bit -c fluent-bit.conf -R parser.conf -q From 42bd1a2e60b8d19eebbda54c0d01d6404daf45cb Mon Sep 17 00:00:00 2001 From: Lynette Miles <6818907+esmerel@users.noreply.github.com> Date: Tue, 1 Jul 2025 13:44:07 -0700 Subject: [PATCH 3/3] Apply suggestions from code review Co-authored-by: Craig Norris <112565517+cnorris-cs@users.noreply.github.com> Signed-off-by: Lynette Miles <6818907+esmerel@users.noreply.github.com> --- pipeline/inputs/standard-input.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/pipeline/inputs/standard-input.md b/pipeline/inputs/standard-input.md index 11baa1bd7..457721c71 100644 --- a/pipeline/inputs/standard-input.md +++ b/pipeline/inputs/standard-input.md @@ -208,7 +208,7 @@ pipeline: {% endtab %} {% endtabs %} -Fluent Bit will now read each line and emit a single message for each input +Fluent Bit will now read each line and emit a single message for each input line, using the following command: ```shell seq 1 5 | /opt/fluent-bit/bin/fluent-bit -c fluent-bit.conf -R parser.conf -q