diff --git a/pipeline/inputs/serial-interface.md b/pipeline/inputs/serial-interface.md index 5a4bf58c7..63492fdb3 100644 --- a/pipeline/inputs/serial-interface.md +++ b/pipeline/inputs/serial-interface.md @@ -24,31 +24,47 @@ To retrieve messages by using the Serial interface, you can run the plugin from The following example loads the input serial plugin where it set a `Bitrate` of `9600`, listens from the `/dev/tnt0` interface, and uses the custom tag `data` to route the message. ```shell -fluent-bit -i serial -t data -p File=/dev/tnt0 -p BitRate=9600 -o stdout -m '*' +$ fluent-bit -i serial -t data -p File=/dev/tnt0 -p BitRate=9600 -o stdout -m '*' ``` The interface (`/dev/tnt0`) is an emulation of the serial interface. Further examples will write some message to the other end of the interface. For example, `/dev/tnt1`. ```shell -echo 'this is some message' > /dev/tnt1 +$ echo 'this is some message' > /dev/tnt1 ``` In Fluent Bit you can run the command: -```bash -fluent-bit -i serial -t data -p File=/dev/tnt0 -p BitRate=9600 -o stdout -m '*' +```shell +$ fluent-bit -i serial -t data -p File=/dev/tnt0 -p BitRate=9600 -o stdout -m '*' ``` Which should produce output like: ```text -Fluent Bit v1.x.x -* Copyright (C) 2019-2020 The Fluent Bit Authors -* Copyright (C) 2015-2018 Treasure Data +Fluent Bit v4.0.3 +* Copyright (C) 2015-2025 The Fluent Bit Authors * Fluent Bit is a CNCF sub-project under the umbrella of Fluentd * https://fluentbit.io -[2016/05/20 15:44:39] [ info] starting engine +______ _ _ ______ _ _ ___ _____ +| ___| | | | | ___ (_) | / || _ | +| |_ | |_ _ ___ _ __ | |_ | |_/ /_| |_ __ __/ /| || |/' | +| _| | | | | |/ _ \ '_ \| __| | ___ \ | __| \ \ / / /_| || /| | +| | | | |_| | __/ | | | |_ | |_/ / | |_ \ V /\___ |\ |_/ / +\_| |_|\__,_|\___|_| |_|\__| \____/|_|\__| \_/ |_(_)___/ + + +[2025/07/01 14:44:47] [ info] [fluent bit] version=4.0.3, commit=f5f5f3c17d, pid=1 +[2025/07/01 14:44:47] [ info] [storage] ver=1.5.3, type=memory, sync=normal, checksum=off, max_chunks_up=128 +[2025/07/01 14:44:47] [ info] [simd ] disabled +[2025/07/01 14:44:47] [ info] [cmetrics] version=1.0.3 +[2025/07/01 14:44:47] [ info] [ctraces ] version=0.6.6 +[2025/07/01 14:44:47] [ info] [input:mem:mem.0] initializing +[2025/07/01 14:44:47] [ info] [input:mem:mem.0] storage_strategy='memory' (memory only) +[2025/07/01 14:44:47] [ info] [sp] stream processor started +[2025/07/01 14:44:47] [ info] [engine] Shutdown Grace Period=5, Shutdown Input Grace Period=2 +[2025/07/01 14:44:47] [ info] [output:stdout:stdout.0] worker #0 started [0] data: [1463780680, {"msg"=>"this is some message"}] ``` @@ -57,22 +73,41 @@ Using the `Separator` configuration, you can send multiple messages at once. Run this command after starting Fluent Bit: ```shell -echo 'aaXbbXccXddXee' > /dev/tnt1 +$ echo 'aaXbbXccXddXee' > /dev/tnt1 ``` Then, run Fluent Bit: ```shell -fluent-bit -i serial -t data -p File=/dev/tnt0 -p BitRate=9600 -p Separator=X -o stdout -m '*' +$ fluent-bit -i serial -t data -p File=/dev/tnt0 -p BitRate=9600 -p Separator=X -o stdout -m '*' ``` This should produce results similar to the following: ```text -Fluent-Bit v0.8.0 -Copyright (C) Treasure Data +Fluent Bit v4.0.3 +* Copyright (C) 2015-2025 The Fluent Bit Authors +* Fluent Bit is a CNCF sub-project under the umbrella of Fluentd +* https://fluentbit.io -[2016/05/20 16:04:51] [ info] starting engine +______ _ _ ______ _ _ ___ _____ +| ___| | | | | ___ (_) | / || _ | +| |_ | |_ _ ___ _ __ | |_ | |_/ /_| |_ __ __/ /| || |/' | +| _| | | | | |/ _ \ '_ \| __| | ___ \ | __| \ \ / / /_| || /| | +| | | | |_| | __/ | | | |_ | |_/ / | |_ \ V /\___ |\ |_/ / +\_| |_|\__,_|\___|_| |_|\__| \____/|_|\__| \_/ |_(_)___/ + + +[2025/07/01 14:44:47] [ info] [fluent bit] version=4.0.3, commit=f5f5f3c17d, pid=1 +[2025/07/01 14:44:47] [ info] [storage] ver=1.5.3, type=memory, sync=normal, checksum=off, max_chunks_up=128 +[2025/07/01 14:44:47] [ info] [simd ] disabled +[2025/07/01 14:44:47] [ info] [cmetrics] version=1.0.3 +[2025/07/01 14:44:47] [ info] [ctraces ] version=0.6.6 +[2025/07/01 14:44:47] [ info] [input:mem:mem.0] initializing +[2025/07/01 14:44:47] [ info] [input:mem:mem.0] storage_strategy='memory' (memory only) +[2025/07/01 14:44:47] [ info] [sp] stream processor started +[2025/07/01 14:44:47] [ info] [engine] Shutdown Grace Period=5, Shutdown Input Grace Period=2 +[2025/07/01 14:44:47] [ info] [output:stdout:stdout.0] worker #0 started [0] data: [1463781902, {"msg"=>"aa"}] [1] data: [1463781902, {"msg"=>"bb"}] [2] data: [1463781902, {"msg"=>"cc"}] @@ -83,7 +118,27 @@ Copyright (C) Treasure Data In your main configuration file append the following sections: -```python +{% tabs %} +{% tab title="fluent-bit.yaml" %} + +```yaml +pipeline: + inputs: + - name: serial + tag: data + file: /dev/tnt0 + bitrate: 9600 + separator: X + + outputs: + - name: stdout + match: '*' +``` + +{% endtab %} +{% tab title="fluent-bit.conf" %} + +```text [INPUT] Name serial Tag data @@ -96,6 +151,9 @@ In your main configuration file append the following sections: Match * ``` +{% endtab %} +{% endtabs %} + ## Emulating a serial interface on Linux You can emulate a serial interface on your Linux system and test the serial input plugin locally when you don't have an interface in your computer. The following procedure has been tested on Ubuntu 15.04 running Linux Kernel 4.0. @@ -104,43 +162,45 @@ You can emulate a serial interface on your Linux system and test the serial inpu 1. Download the sources: - ```bash - git clone https://github.com/freemed/tty0tty + ```shell + $ git clone https://github.com/freemed/tty0tty ``` -1. Unpack and compile: +2. Unpack and compile: - ```bash - cd tty0tty/module - make + ```shell + $ cd tty0tty/module + + $ make ``` -1. Copy the new kernel module into the kernel modules directory: +3. Copy the new kernel module into the kernel modules directory: - ```bash - sudo cp tty0tty.ko /lib/modules/$(uname -r)/kernel/drivers/misc/ + ```shell + $ sudo cp tty0tty.ko /lib/modules/$(uname -r)/kernel/drivers/misc/ ``` -1. Load the module: +4. Load the module: - ```bash - sudo depmod - sudo modprobe tty0tty + ```shell + $ sudo depmod + + $ sudo modprobe tty0tty ``` You should see new serial ports in `dev` (`ls /dev/tnt\*\`). -1. Give appropriate permissions to the new serial ports: +5. Give appropriate permissions to the new serial ports: - ```bash - sudo chmod 666 /dev/tnt* + ```shell + $ sudo chmod 666 /dev/tnt* ``` When the module is loaded, it will interconnect the following virtual interfaces: -```bash +```text /dev/tnt0 <=> /dev/tnt1 /dev/tnt2 <=> /dev/tnt3 /dev/tnt4 <=> /dev/tnt5 /dev/tnt6 <=> /dev/tnt7 -``` +``` \ No newline at end of file diff --git a/pipeline/inputs/splunk.md b/pipeline/inputs/splunk.md index 2a35473ed..60fe042e3 100644 --- a/pipeline/inputs/splunk.md +++ b/pipeline/inputs/splunk.md @@ -41,15 +41,33 @@ To use the other tags for multiple instantiating input Splunk plugins, you must From the command line you can configure Fluent Bit to handle HTTP HEC requests with the following options: -```bash -fluent-bit -i splunk -p port=8088 -o stdout +```shell +$ fluent-bit -i splunk -p port=8088 -o stdout ``` ### Configuration file In your main configuration file append the following sections: -```python +{% tabs %} +{% tab title="fluent-bit.yaml" %} + +```yaml +pipeline: + inputs: + - name: splunk + listen: 0.0.0.0 + port: 8088 + + outputs: + - name: stdout + match: '*' +``` + +{% endtab %} +{% tab title="fluent-bit.conf" %} + +```text [INPUT] name splunk listen 0.0.0.0 @@ -59,3 +77,6 @@ In your main configuration file append the following sections: name stdout match * ``` + +{% endtab %} +{% endtabs %} \ No newline at end of file diff --git a/pipeline/inputs/standard-input.md b/pipeline/inputs/standard-input.md index 457721c71..bd960cc0f 100644 --- a/pipeline/inputs/standard-input.md +++ b/pipeline/inputs/standard-input.md @@ -3,8 +3,8 @@ The _Standard input_ plugin supports retrieving a message stream from the standard input interface (`stdin`) of the Fluent Bit process. To use it, specify the plugin name as the input. For example: -```bash - fluent-bit -i stdin -o stdout +```shell +$ fluent-bit -i stdin -o stdout ``` If the `stdin` stream is closed (`end-of-file`), the plugin instructs Fluent Bit to exit with success (`0`) after flushing any pending output. @@ -60,110 +60,124 @@ To demonstrate how the plugin works, you can use a `bash` script that generates done ``` -1. Start the script and [Fluent Bit](http://fluentbit.io): +2. Start the script and [Fluent Bit](http://fluentbit.io): - ```bash - bash test.sh | fluent-bit -q -i stdin -o stdout + ```shell + $ bash test.sh | fluent-bit -q -i stdin -o stdout ``` -The command should return output like the following: +3. The command should return output like the following: -```text -[0] stdin.0: [[1684196745.942883835, {}], {"key"=>"some value"}] -[0] stdin.0: [[1684196746.938949056, {}], {"key"=>"some value"}] -[0] stdin.0: [[1684196747.940162493, {}], {"key"=>"some value"}] -[0] stdin.0: [[1684196748.941392297, {}], {"key"=>"some value"}] -[0] stdin.0: [[1684196749.942644238, {}], {"key"=>"some value"}] -[0] stdin.0: [[1684196750.943721442, {}], {"key"=>"some value"}] -``` + ```shell + [0] stdin.0: [[1684196745.942883835, {}], {"key"=>"some value"}] + [0] stdin.0: [[1684196746.938949056, {}], {"key"=>"some value"}] + [0] stdin.0: [[1684196747.940162493, {}], {"key"=>"some value"}] + [0] stdin.0: [[1684196748.941392297, {}], {"key"=>"some value"}] + [0] stdin.0: [[1684196749.942644238, {}], {"key"=>"some value"}] + [0] stdin.0: [[1684196750.943721442, {}], {"key"=>"some value"}] + ``` ### JSON input with timestamp -An input event timestamp can also be supplied. Replace `test.sh` with: - -```bash -#!/bin/sh - -for ((i=0; i<=5; i++)); do - echo -n " - [ - $(date '+%s.%N' -d '1 day ago'), - { - \"realtimestamp\": $(date '+%s.%N') - } - ] - " - sleep 1 -done -``` - -Re-run the sample command. Timestamps output by Fluent Bit are now one day old because Fluent Bit used the input message timestamp. - -```bash -bash test.sh | fluent-bit -q -i stdin -o stdout -``` - -Which returns the following: - -```text -[0] stdin.0: [[1684110480.028171300, {}], {"realtimestamp"=>1684196880.030070}] -[0] stdin.0: [[1684110481.033753395, {}], {"realtimestamp"=>1684196881.034741}] -[0] stdin.0: [[1684110482.036730051, {}], {"realtimestamp"=>1684196882.037704}] -[0] stdin.0: [[1684110483.039903879, {}], {"realtimestamp"=>1684196883.041081}] -[0] stdin.0: [[1684110484.044719457, {}], {"realtimestamp"=>1684196884.046404}] -[0] stdin.0: [[1684110485.048710107, {}], {"realtimestamp"=>1684196885.049651}] -``` +1. An input event timestamp can also be supplied. Replace `test.sh` with: + + ```bash + #!/bin/sh + + for ((i=0; i<=5; i++)); do + echo -n " + [ + $(date '+%s.%N' -d '1 day ago'), + { + \"realtimestamp\": $(date '+%s.%N') + } + ] + " + sleep 1 + done + ``` + +2. Re-run the sample command. Timestamps output by Fluent Bit are now one day old because Fluent Bit used the input message timestamp. + + ```shell + $ bash test.sh | fluent-bit -q -i stdin -o stdout + ``` + +3. Which returns the following: + + ```shell + [0] stdin.0: [[1684110480.028171300, {}], {"realtimestamp"=>1684196880.030070}] + [0] stdin.0: [[1684110481.033753395, {}], {"realtimestamp"=>1684196881.034741}] + [0] stdin.0: [[1684110482.036730051, {}], {"realtimestamp"=>1684196882.037704}] + [0] stdin.0: [[1684110483.039903879, {}], {"realtimestamp"=>1684196883.041081}] + [0] stdin.0: [[1684110484.044719457, {}], {"realtimestamp"=>1684196884.046404}] + [0] stdin.0: [[1684110485.048710107, {}], {"realtimestamp"=>1684196885.049651}] + ``` ### JSON input with metadata -Additional metadata is supported in Fluent Bit v2.1.0 and later by replacing the timestamp with a two-element object. For example: - -```bash -#!/bin/sh -for ((i=0; i<=5; i++)); do - echo -n " - [ - [ - $(date '+%s.%N' -d '1 day ago'), - {\"metakey\": \"metavalue\"} - ], - { - \"realtimestamp\": $(date '+%s.%N') - } - ] - " - sleep 1 -done -``` +1. Additional metadata is supported in Fluent Bit v2.1.0 and later by replacing the timestamp with a two-element object. For example: + + ```bash + #!/bin/sh + for ((i=0; i<=5; i++)); do + echo -n " + [ + [ + $(date '+%s.%N' -d '1 day ago'), + {\"metakey\": \"metavalue\"} + ], + { + \"realtimestamp\": $(date '+%s.%N') + } + ] + " + sleep 1 + done + ``` + +2. Run test using the command: + + ```shell + $ bash ./test.sh | fluent-bit -q -i stdin -o stdout + ``` + +3. Which returns results like the following: + + ```shell + [0] stdin.0: [[1684110513.060139417, {"metakey"=>"metavalue"}], {"realtimestamp"=>1684196913.061017}] + [0] stdin.0: [[1684110514.063085317, {"metakey"=>"metavalue"}], {"realtimestamp"=>1684196914.064145}] + [0] stdin.0: [[1684110515.066210508, {"metakey"=>"metavalue"}], {"realtimestamp"=>1684196915.067155}] + [0] stdin.0: [[1684110516.069149971, {"metakey"=>"metavalue"}], {"realtimestamp"=>1684196916.070132}] + [0] stdin.0: [[1684110517.072484016, {"metakey"=>"metavalue"}], {"realtimestamp"=>1684196917.073636}] + [0] stdin.0: [[1684110518.075428724, {"metakey"=>"metavalue"}], {"realtimestamp"=>1684196918.076292}] + ``` + +4. On older Fluent Bit versions records in this format will be discarded. If the log level permits, Fluent Bit will log: + + ```shell + [ warn] unknown time format 6 + ``` -Run test using the command: +### Parser input -```bash -bash ./test.sh | fluent-bit -q -i stdin -o stdout -``` +To capture inputs in other formats, specify a parser configuration for the `stdin` plugin. -Which returns results like the following: +For example, if you want to read raw messages line by line and forward them, you could use a separate parsers file that captures the whole message line: -```text -[0] stdin.0: [[1684110513.060139417, {"metakey"=>"metavalue"}], {"realtimestamp"=>1684196913.061017}] -[0] stdin.0: [[1684110514.063085317, {"metakey"=>"metavalue"}], {"realtimestamp"=>1684196914.064145}] -[0] stdin.0: [[1684110515.066210508, {"metakey"=>"metavalue"}], {"realtimestamp"=>1684196915.067155}] -[0] stdin.0: [[1684110516.069149971, {"metakey"=>"metavalue"}], {"realtimestamp"=>1684196916.070132}] -[0] stdin.0: [[1684110517.072484016, {"metakey"=>"metavalue"}], {"realtimestamp"=>1684196917.073636}] -[0] stdin.0: [[1684110518.075428724, {"metakey"=>"metavalue"}], {"realtimestamp"=>1684196918.076292}] -``` - -On older Fluent Bit versions records in this format will be discarded. If the log level permits, Fluent Bit will log: +{% tabs %} +{% tab title="parsers.yaml" %} -```text -[ warn] unknown time format 6 +```yaml +parsers: + - name: stringify_message + format: regex + key_name: message + regex: '^(?.*)' ``` -### Parser input - -To capture inputs in other formats, specify a parser configuration for the `stdin` plugin. - -For example, if you want to read raw messages line by line and forward them, you could use a `parser.conf` that captures the whole message line: +{% endtab %} +{% tab title="parsers.conf" %} ```text [PARSER] @@ -173,28 +187,36 @@ For example, if you want to read raw messages line by line and forward them, you regex ^(?.*) ``` -You can then use that in the `parser` clause of the `stdin` plugin in the `fluent-bit.conf` file: +{% endtab %} +{% endtabs %} + +You can then use the parsers file in a `stdin` plugin in the main Fluent Bit configuration file as follows: {% tabs %} - {% tab title="fluent-bit.yaml" %} ```yaml +service: + parsers_file: parsers.yaml + pipeline: inputs: - name: stdin tag: stdin parser: stringify_message + outputs: - name: stdout match: '*' ``` {% endtab %} - {% tab title="fluent-bit.conf" %} ```text +[SERVICE] + parsers_file parsers.conf + [INPUT] Name stdin Tag stdin @@ -211,18 +233,46 @@ pipeline: Fluent Bit will now read each line and emit a single message for each input line, using the following command: ```shell -seq 1 5 | /opt/fluent-bit/bin/fluent-bit -c fluent-bit.conf -R parser.conf -q +# For YAML configuration. +$ seq 1 5 | ./fluent-bit --config fluent-bit.yaml + +# For classic configuration. +$ seq 1 5 | ./fluent-bit --config fluent-bit.conf ``` Which returns output similar to: ```text -[0] stdin: [1681358780.517029169, {"message"=>"1"}] -[1] stdin: [1681358780.517068334, {"message"=>"2"}] -[2] stdin: [1681358780.517072116, {"message"=>"3"}] -[3] stdin: [1681358780.517074758, {"message"=>"4"}] -[4] stdin: [1681358780.517077392, {"message"=>"5"}] -$ +Fluent Bit v4.0.3 +* Copyright (C) 2015-2025 The Fluent Bit Authors +* Fluent Bit is a CNCF sub-project under the umbrella of Fluentd +* https://fluentbit.io + +______ _ _ ______ _ _ ___ _____ +| ___| | | | | ___ (_) | / || _ | +| |_ | |_ _ ___ _ __ | |_ | |_/ /_| |_ __ __/ /| || |/' | +| _| | | | | |/ _ \ '_ \| __| | ___ \ | __| \ \ / / /_| || /| | +| | | | |_| | __/ | | | |_ | |_/ / | |_ \ V /\___ |\ |_/ / +\_| |_|\__,_|\___|_| |_|\__| \____/|_|\__| \_/ |_(_)___/ + + +[2025/07/03 14:32:54] [ info] [fluent bit] version=4.0.3, commit=3a91b155d6, pid=18569 +[2025/07/03 14:32:54] [ info] [storage] ver=1.5.3, type=memory, sync=normal, checksum=off, max_chunks_up=128 +[2025/07/03 14:32:54] [ info] [simd ] disabled +[2025/07/03 14:32:54] [ info] [cmetrics] version=1.0.3 +[2025/07/03 14:32:54] [ info] [ctraces ] version=0.6.6 +[2025/07/03 14:32:54] [ info] [input:stdin:stdin.0] initializing +[2025/07/03 14:32:54] [ info] [input:stdin:stdin.0] storage_strategy='memory' (memory only) +[2025/07/03 14:32:54] [ info] [sp] stream processor started +[2025/07/03 14:32:54] [ info] [output:stdout:stdout.0] worker #0 started +[2025/07/03 14:32:54] [ info] [engine] Shutdown Grace Period=5, Shutdown Input Grace Period=2 +[2025/07/03 14:32:54] [ warn] [input:stdin:stdin.0] end of file (stdin closed by remote end) +[2025/07/03 14:32:54] [ warn] [engine] service will shutdown in max 5 seconds +[0] stdin: [[1751545974.960182000, {}], {"message"=>"1"}] +[1] stdin: [[1751545974.960246000, {}], {"message"=>"2"}] +[2] stdin: [[1751545974.960255000, {}], {"message"=>"3"}] +[3] stdin: [[1751545974.960262000, {}], {"message"=>"4"}] +[4] stdin: [[1751545974.960268000, {}], {"message"=>"5"}] ``` -In production deployments it's best to use a parser that splits messages into real fields and adds appropriate tags. +In production deployments it's best to use a parser that splits messages into real fields and adds appropriate tags. \ No newline at end of file diff --git a/pipeline/inputs/statsd.md b/pipeline/inputs/statsd.md index 2f3d28926..9252f48cd 100644 --- a/pipeline/inputs/statsd.md +++ b/pipeline/inputs/statsd.md @@ -35,6 +35,7 @@ pipeline: - name: statsd listen: 0.0.0.0 port: 8125 + outputs: - name: stdout match: '*' @@ -43,7 +44,7 @@ pipeline: {% endtab %} {% tab title="fluent-bit.conf" %} -```python +```text [INPUT] Name statsd Listen 0.0.0.0 @@ -59,9 +60,10 @@ pipeline: Now you can input metrics through the UDP port as follows: -```bash -echo "click:10|c|@0.1" | nc -q0 -u 127.0.0.1 8125 -echo "active:99|g" | nc -q0 -u 127.0.0.1 8125 +```shell +$ echo "click:10|c|@0.1" | nc -q0 -u 127.0.0.1 8125 + +$ echo "active:99|g" | nc -q0 -u 127.0.0.1 8125 ``` Fluent Bit will produce the following records: @@ -76,7 +78,6 @@ Fluent Bit will produce the following records: Here is a configuration example for metrics setup. {% tabs %} - {% tab title="fluent-bit.yaml" %} ```yaml @@ -86,6 +87,7 @@ pipeline: listen: 0.0.0.0 port: 8125 metrics: On + outputs: - name: stdout match: '*' @@ -94,7 +96,7 @@ pipeline: {% endtab %} {% tab title="fluent-bit.conf" %} -```python +```text [INPUT] Name statsd Listen 0.0.0.0 @@ -111,10 +113,12 @@ pipeline: Now you can input metrics as metrics type of events through the UDP port as follows: -```bash -echo "click:+10|c|@0.01|#hello:tag" | nc -q0 -u 127.0.0.1 8125 -echo "active:+99|g|@0.01" | nc -q0 -u 127.0.0.1 8125 -echo "inactive:29|g|@0.0125|#hi:from_fluent-bit" | nc -q0 -u 127.0.0.1 8125 +```shell +$ echo "click:+10|c|@0.01|#hello:tag" | nc -q0 -u 127.0.0.1 8125 + +$ echo "active:+99|g|@0.01" | nc -q0 -u 127.0.0.1 8125 + +$ echo "inactive:29|g|@0.0125|#hi:from_fluent-bit" | nc -q0 -u 127.0.0.1 8125 ``` Fluent Bit will produce the following metrics events: @@ -123,4 +127,4 @@ Fluent Bit will produce the following metrics events: 2025-01-09T11:40:26.562424694Z click{incremental="true",hello="tag"} = 1000 2025-01-09T11:40:28.591477424Z active{incremental="true"} = 9900 2025-01-09T11:40:31.593118033Z inactive{hi="from_fluent-bit"} = 2320 -``` +``` \ No newline at end of file diff --git a/pipeline/inputs/syslog.md b/pipeline/inputs/syslog.md index 0ee446cbb..ef721c070 100644 --- a/pipeline/inputs/syslog.md +++ b/pipeline/inputs/syslog.md @@ -22,8 +22,8 @@ The plugin supports the following configuration parameters: ### Considerations -- When using the Syslog input plugin, Fluent Bit requires access to the `parsers.conf` file. The path to this file can be specified with the option `-R` or through the `Parsers_File` key in the `[SERVICE]` section. -- When using `udp` or `unix_udp`, the buffer size to receive messages is configurable only through the `Buffer_Chunk_Size` option, which defaults to 32kb. +- When using the Syslog input plugin, Fluent Bit requires access to the parsers configuration file. The path to this file can be specified with the option `-R` or through the `parsers_file` key in the service section. +- When using `udp` or `unix_udp`, the buffer size to receive messages is configurable only through the `buffer_chunk_size` option, which defaults to 32kb. ## Get started @@ -33,8 +33,12 @@ To receive `syslog` messages, you can run the plugin from the command line or th From the command line you can let Fluent Bit listen for `Forward` messages with the following options: -```bash -fluent-bit -R /path/to/parsers.conf -i syslog -p path=/tmp/in_syslog -o stdout +```shell +# For YAML configuration +$ ./fluent-bit -R /path/to/parsers.yaml -i syslog -p path=/tmp/in_syslog -o stdout + +# For classic configuration. +$ ./fluent-bit -R /path/to/parsers.conf -i syslog -p path=/tmp/in_syslog -o stdout ``` By default the service will create and listen for Syslog messages on the Unix socket `/tmp/in_syslog`. @@ -44,14 +48,14 @@ By default the service will create and listen for Syslog messages on the Unix so In your main configuration file append the following sections: {% tabs %} - {% tab title="fluent-bit.yaml" %} ```yaml service: flush: 1 log_level: info - parsers_file: parsers.conf + parsers_file: parsers.yaml + pipeline: inputs: - name: syslog @@ -59,6 +63,7 @@ pipeline: buffer_chunk_size: 32000 buffer_max_size: 64000 receive_buffer_size: 512000 + outputs: - name: stdout match: '*' @@ -66,7 +71,8 @@ pipeline: {% endtab %} {% tab title="fluent-bit.conf" %} -```python + +```text [SERVICE] Flush 1 Log_Level info @@ -85,33 +91,52 @@ pipeline: ``` {% endtab %} - {% endtabs %} ### Testing When Fluent Bit is running, you can send some messages using the logger tool: -```bash -logger -u /tmp/in_syslog my_ident my_message +```shell +$ logger -u /tmp/in_syslog my_ident my_message ``` Then run Fluent bit using the following command: -```bash -bin/fluent-bit -R ../conf/parsers.conf -i syslog -p path=/tmp/in_syslog -o stdout +```shell +# For YAML ocnfiguration. +$ ./fluent-bit -R ../conf/parsers.yaml -i syslog -p path=/tmp/in_syslog -o stdout + +# For classic configuration. +$ ./fluent-bit -R ../conf/parsers.conf -i syslog -p path=/tmp/in_syslog -o stdout ``` You should see the following output: ```text -Fluent Bit v1.x.x -* Copyright (C) 2019-2020 The Fluent Bit Authors -* Copyright (C) 2015-2018 Treasure Data +Fluent Bit v4.0.3 +* Copyright (C) 2015-2025 The Fluent Bit Authors * Fluent Bit is a CNCF sub-project under the umbrella of Fluentd * https://fluentbit.io -[2017/03/09 02:23:27] [ info] [engine] started +______ _ _ ______ _ _ ___ _____ +| ___| | | | | ___ (_) | / || _ | +| |_ | |_ _ ___ _ __ | |_ | |_/ /_| |_ __ __/ /| || |/' | +| _| | | | | |/ _ \ '_ \| __| | ___ \ | __| \ \ / / /_| || /| | +| | | | |_| | __/ | | | |_ | |_/ / | |_ \ V /\___ |\ |_/ / +\_| |_|\__,_|\___|_| |_|\__| \____/|_|\__| \_/ |_(_)___/ + + +[2025/07/01 14:44:47] [ info] [fluent bit] version=4.0.3, commit=f5f5f3c17d, pid=1 +[2025/07/01 14:44:47] [ info] [storage] ver=1.5.3, type=memory, sync=normal, checksum=off, max_chunks_up=128 +[2025/07/01 14:44:47] [ info] [simd ] disabled +[2025/07/01 14:44:47] [ info] [cmetrics] version=1.0.3 +[2025/07/01 14:44:47] [ info] [ctraces ] version=0.6.6 +[2025/07/01 14:44:47] [ info] [input:mem:mem.0] initializing +[2025/07/01 14:44:47] [ info] [input:mem:mem.0] storage_strategy='memory' (memory only) +[2025/07/01 14:44:47] [ info] [sp] stream processor started +[2025/07/01 14:44:47] [ info] [engine] Shutdown Grace Period=5, Shutdown Input Grace Period=2 +[2025/07/01 14:44:47] [ info] [output:stdout:stdout.0] worker #0 started [0] syslog.0: [1489047822, {"pri"=>"13", "host"=>"edsiper:", "ident"=>"my_ident", "pid"=>"", "message"=>"my_message"}] ``` @@ -131,7 +156,8 @@ Put the following content in your configuration file: ```yaml service: flush: 1 - parsers_file: parsers.conf + parsers_file: parsers.yaml + pipeline: inputs: - name: syslog @@ -139,13 +165,13 @@ pipeline: listen: 0.0.0.0 port: 5140 mode: tcp + outputs: - name: stdout match: '*' ``` {% endtab %} - {% tab title="fluent-bit.conf" %} ```text @@ -180,15 +206,15 @@ action(type="omfwd" Target="127.0.0.1" Port="5140" Protocol="tcp") Then, restart your `rsyslog` daemon: -```bash -sudo service rsyslog restart +```shell +$ sudo service rsyslog restart ``` ### `rsyslog` to Fluent Bit: Unix socket mode over UDP #### Fluent Bit configuration -Put the following content in your `fluent-bit.conf` file: +Put the following content in your Fluent Bit configuration: {% tabs %} {% tab title="fluent-bit.yaml" %} @@ -196,7 +222,8 @@ Put the following content in your `fluent-bit.conf` file: ```yaml service: flush: 1 - parsers_file: parsers.conf + parsers_file: parsers.yaml + pipeline: inputs: - name: syslog @@ -204,6 +231,7 @@ pipeline: path: /tmp/fluent-bit.sock mode: unix_udp unix_perm: 0644 + outputs: - name: stdout match: '*' @@ -244,4 +272,4 @@ $OMUxSockSocket /tmp/fluent-bit.sock *.* :omuxsock: ``` -Make sure that the socket file is readable by `rsyslog` by modifying `Unix_Perm` key. +Make sure that the socket file is readable by `rsyslog` by modifying `Unix_Perm` key. \ No newline at end of file diff --git a/pipeline/inputs/systemd.md b/pipeline/inputs/systemd.md index 36066b090..02a5d0a5c 100644 --- a/pipeline/inputs/systemd.md +++ b/pipeline/inputs/systemd.md @@ -29,7 +29,7 @@ In order to receive Systemd messages, you can run the plugin from the command li From the command line you can let Fluent Bit listen for _Systemd_ messages with the following options: -```bash +```shell $ fluent-bit -i systemd \ -p systemd_filter=_SYSTEMD_UNIT=docker.service \ -p tag='host.*' -o stdout @@ -42,7 +42,27 @@ $ fluent-bit -i systemd \ In your main configuration file append the following _Input_ & _Output_ sections: {% tabs %} +{% tab title="fluent-bit.yaml" %} + +```yaml +service: + flush: 1 + log_level: info + parsers_file: parsers.yaml + +pipeline: + inputs: + - name: systemd + tag: host.* + systemd_filter: _SYSTEMD_UNIT=docker.service + outputs: + - name: stdout + match: '*' +``` + +{% endtab %} {% tab title="fluent-bit.conf" %} + ```text [SERVICE] Flush 1 @@ -58,22 +78,6 @@ In your main configuration file append the following _Input_ & _Output_ sections Name stdout Match * ``` -{% endtab %} -{% tab title="fluent-bit.yaml" %} -```yaml -service: - flush: 1 - log_level: info - parsers_file: parsers.conf -pipeline: - inputs: - - name: systemd - tag: host.* - systemd_filter: _SYSTEMD_UNIT=docker.service - outputs: - - name: stdout - match: '*' -``` {% endtab %} -{% endtabs %} +{% endtabs %} \ No newline at end of file diff --git a/pipeline/inputs/tail.md b/pipeline/inputs/tail.md index f0d58dc49..5335b57ec 100644 --- a/pipeline/inputs/tail.md +++ b/pipeline/inputs/tail.md @@ -65,6 +65,7 @@ Replace _`LIMIT1`_ and _`LIMIT2`_ with the integer values of your choosing. You can also provide a custom systemd configuration file that overrides the default systemd settings for Fluent Bit. This override file must be located at `/etc/systemd/system/fluent-bit.service.d/override.conf` or `/etc/systemd/system/fluent-bit.service.d/override.yaml` depending on the configuration you choose. For example, you can add one of these snippets to your override file to raise the number of files that the Tail plugin can monitor: + {% tabs %} {% tab title="override.yaml" %} diff --git a/pipeline/inputs/tcp.md b/pipeline/inputs/tcp.md index 67dba0eb5..30ace3dd9 100644 --- a/pipeline/inputs/tcp.md +++ b/pipeline/inputs/tcp.md @@ -25,13 +25,13 @@ In order to receive JSON messages over TCP, you can run the plugin from the comm From the command line you can let Fluent Bit listen for _JSON_ messages with the following options: -```bash +```shell $ fluent-bit -i tcp -o stdout ``` By default the service will listen an all interfaces (0.0.0.0) through TCP port 5170, optionally you can change this directly, e.g: -```bash +```shell $ fluent-bit -i tcp://192.168.3.2:9090 -o stdout ``` @@ -42,23 +42,8 @@ In the example the JSON messages will only arrive through network interface unde In your main configuration file append the following _Input_ & _Output_ sections: {% tabs %} -{% tab title="fluent-bit.conf" %} -```python -[INPUT] - Name tcp - Listen 0.0.0.0 - Port 5170 - Chunk_Size 32 - Buffer_Size 64 - Format json - -[OUTPUT] - Name stdout - Match * -``` -{% endtab %} - {% tab title="fluent-bit.yaml" %} + ```yaml pipeline: inputs: @@ -68,10 +53,29 @@ pipeline: chunk_size: 32 buffer_size: 64 format: json + outputs: - name: stdout match: '*' ``` + +{% endtab %} +{% tab title="fluent-bit.conf" %} + +```text +[INPUT] + Name tcp + Listen 0.0.0.0 + Port 5170 + Chunk_Size 32 + Buffer_Size 64 + Format json + +[OUTPUT] + Name stdout + Match * +``` + {% endtab %} {% endtabs %} @@ -79,25 +83,38 @@ pipeline: Once Fluent Bit is running, you can send some messages using the _netcat_: -```bash +```shell $ echo '{"key 1": 123456789, "key 2": "abcdefg"}' | nc 127.0.0.1 5170 ``` In [Fluent Bit](http://fluentbit.io) we should see the following output: -```bash +```shell $ bin/fluent-bit -i tcp -o stdout -f 1 -Fluent Bit v1.x.x -* Copyright (C) 2019-2020 The Fluent Bit Authors -* Copyright (C) 2015-2018 Treasure Data + +Fluent Bit v4.0.3 +* Copyright (C) 2015-2025 The Fluent Bit Authors * Fluent Bit is a CNCF sub-project under the umbrella of Fluentd * https://fluentbit.io -[2019/10/03 09:19:34] [ info] [storage] initializing... -[2019/10/03 09:19:34] [ info] [storage] in-memory -[2019/10/03 09:19:34] [ info] [engine] started (pid=14569) -[2019/10/03 09:19:34] [ info] [in_tcp] binding 0.0.0.0:5170 -[2019/10/03 09:19:34] [ info] [sp] stream processor started +______ _ _ ______ _ _ ___ _____ +| ___| | | | | ___ (_) | / || _ | +| |_ | |_ _ ___ _ __ | |_ | |_/ /_| |_ __ __/ /| || |/' | +| _| | | | | |/ _ \ '_ \| __| | ___ \ | __| \ \ / / /_| || /| | +| | | | |_| | __/ | | | |_ | |_/ / | |_ \ V /\___ |\ |_/ / +\_| |_|\__,_|\___|_| |_|\__| \____/|_|\__| \_/ |_(_)___/ + + +[2025/07/01 14:44:47] [ info] [fluent bit] version=4.0.3, commit=f5f5f3c17d, pid=1 +[2025/07/01 14:44:47] [ info] [storage] ver=1.5.3, type=memory, sync=normal, checksum=off, max_chunks_up=128 +[2025/07/01 14:44:47] [ info] [simd ] disabled +[2025/07/01 14:44:47] [ info] [cmetrics] version=1.0.3 +[2025/07/01 14:44:47] [ info] [ctraces ] version=0.6.6 +[2025/07/01 14:44:47] [ info] [input:mem:mem.0] initializing +[2025/07/01 14:44:47] [ info] [input:mem:mem.0] storage_strategy='memory' (memory only) +[2025/07/01 14:44:47] [ info] [sp] stream processor started +[2025/07/01 14:44:47] [ info] [engine] Shutdown Grace Period=5, Shutdown Input Grace Period=2 +[2025/07/01 14:44:47] [ info] [output:stdout:stdout.0] worker #0 started [0] tcp.0: [1570115975.581246030, {"key 1"=>123456789, "key 2"=>"abcdefg"}] ``` @@ -105,4 +122,4 @@ Fluent Bit v1.x.x When receiving payloads in JSON format, there are high performance penalties. Parsing JSON is a very expensive task so you could expect your CPU usage increase under high load environments. -To get faster data ingestion, consider to use the option `Format none` to avoid JSON parsing if not needed. +To get faster data ingestion, consider to use the option `Format none` to avoid JSON parsing if not needed. \ No newline at end of file diff --git a/pipeline/inputs/thermal.md b/pipeline/inputs/thermal.md index 56af07975..406397a11 100644 --- a/pipeline/inputs/thermal.md +++ b/pipeline/inputs/thermal.md @@ -1,6 +1,6 @@ # Thermal -The **thermal** input plugin reports system temperatures periodically -- each second by default. Currently this plugin is only available for Linux. +The **thermal** input plugin reports system temperatures periodically -- each second by default. Currently, this plugin is only available for Linux. The following tables describes the information generated by the plugin. @@ -28,16 +28,32 @@ In order to get temperature\(s\) of your system, you can run the plugin from the ### Command Line -```bash -$ bin/fluent-bit -i thermal -t my_thermal -o stdout -m '*' -Fluent Bit v1.x.x -* Copyright (C) 2019-2020 The Fluent Bit Authors -* Copyright (C) 2015-2018 Treasure Data +```shell +$ ./fluent-bit -i thermal -t my_thermal -o stdout -m '*' + +Fluent Bit v4.0.3 +* Copyright (C) 2015-2025 The Fluent Bit Authors * Fluent Bit is a CNCF sub-project under the umbrella of Fluentd * https://fluentbit.io -[2019/08/18 13:39:43] [ info] [storage] initializing... -... +______ _ _ ______ _ _ ___ _____ +| ___| | | | | ___ (_) | / || _ | +| |_ | |_ _ ___ _ __ | |_ | |_/ /_| |_ __ __/ /| || |/' | +| _| | | | | |/ _ \ '_ \| __| | ___ \ | __| \ \ / / /_| || /| | +| | | | |_| | __/ | | | |_ | |_/ / | |_ \ V /\___ |\ |_/ / +\_| |_|\__,_|\___|_| |_|\__| \____/|_|\__| \_/ |_(_)___/ + + +[2025/07/01 14:44:47] [ info] [fluent bit] version=4.0.3, commit=f5f5f3c17d, pid=1 +[2025/07/01 14:44:47] [ info] [storage] ver=1.5.3, type=memory, sync=normal, checksum=off, max_chunks_up=128 +[2025/07/01 14:44:47] [ info] [simd ] disabled +[2025/07/01 14:44:47] [ info] [cmetrics] version=1.0.3 +[2025/07/01 14:44:47] [ info] [ctraces ] version=0.6.6 +[2025/07/01 14:44:47] [ info] [input:mem:mem.0] initializing +[2025/07/01 14:44:47] [ info] [input:mem:mem.0] storage_strategy='memory' (memory only) +[2025/07/01 14:44:47] [ info] [sp] stream processor started +[2025/07/01 14:44:47] [ info] [engine] Shutdown Grace Period=5, Shutdown Input Grace Period=2 +[2025/07/01 14:44:47] [ info] [output:stdout:stdout.0] worker #0 started [0] my_thermal: [1566099584.000085820, {"name"=>"thermal_zone0", "type"=>"x86_pkg_temp", "temp"=>60.000000}] [1] my_thermal: [1566099585.000136466, {"name"=>"thermal_zone0", "type"=>"x86_pkg_temp", "temp"=>59.000000}] [2] my_thermal: [1566099586.000083156, {"name"=>"thermal_zone0", "type"=>"x86_pkg_temp", "temp"=>59.000000}] @@ -45,43 +61,66 @@ Fluent Bit v1.x.x Some systems provide multiple thermal zones. In this example monitor only _thermal\_zone0_ by name, once per minute. -```bash -$ bin/fluent-bit -i thermal -t my_thermal -p "interval_sec=60" -p "name_regex=thermal_zone0" -o stdout -m '*' -Fluent Bit v1.3.0 -Copyright (C) Treasure Data +```shell +$ ./fluent-bit -i thermal -t my_thermal -p "interval_sec=60" -p "name_regex=thermal_zone0" -o stdout -m '*' + +Fluent Bit v4.0.3 +* Copyright (C) 2015-2025 The Fluent Bit Authors +* Fluent Bit is a CNCF sub-project under the umbrella of Fluentd +* https://fluentbit.io -[2019/08/18 13:39:43] [ info] [storage] initializing... -... +______ _ _ ______ _ _ ___ _____ +| ___| | | | | ___ (_) | / || _ | +| |_ | |_ _ ___ _ __ | |_ | |_/ /_| |_ __ __/ /| || |/' | +| _| | | | | |/ _ \ '_ \| __| | ___ \ | __| \ \ / / /_| || /| | +| | | | |_| | __/ | | | |_ | |_/ / | |_ \ V /\___ |\ |_/ / +\_| |_|\__,_|\___|_| |_|\__| \____/|_|\__| \_/ |_(_)___/ + + +[2025/07/01 14:44:47] [ info] [fluent bit] version=4.0.3, commit=f5f5f3c17d, pid=1 +[2025/07/01 14:44:47] [ info] [storage] ver=1.5.3, type=memory, sync=normal, checksum=off, max_chunks_up=128 +[2025/07/01 14:44:47] [ info] [simd ] disabled +[2025/07/01 14:44:47] [ info] [cmetrics] version=1.0.3 +[2025/07/01 14:44:47] [ info] [ctraces ] version=0.6.6 +[2025/07/01 14:44:47] [ info] [input:mem:mem.0] initializing +[2025/07/01 14:44:47] [ info] [input:mem:mem.0] storage_strategy='memory' (memory only) +[2025/07/01 14:44:47] [ info] [sp] stream processor started +[2025/07/01 14:44:47] [ info] [engine] Shutdown Grace Period=5, Shutdown Input Grace Period=2 +[2025/07/01 14:44:47] [ info] [output:stdout:stdout.0] worker #0 started [0] my_temp: [1565759542.001053749, {"name"=>"thermal_zone0", "type"=>"pch_skylake", "temp"=>48.500000}] [0] my_temp: [1565759602.001661061, {"name"=>"thermal_zone0", "type"=>"pch_skylake", "temp"=>48.500000}] ``` ### Configuration File -In your main configuration file append the following _Input_ & _Output_ sections: +In your main configuration file append the following: {% tabs %} -{% tab title="fluent-bit.conf" %} -```python -[INPUT] - Name thermal - Tag my_thermal - -[OUTPUT] - Name stdout - Match * -``` -{% endtab %} - {% tab title="fluent-bit.yaml" %} + ```yaml pipeline: inputs: - name: thermal tag: my_thermal + outputs: - name: stdout match: '*' ``` + +{% endtab %} +{% tab title="fluent-bit.conf" %} + +```text +[INPUT] + Name thermal + Tag my_thermal + +[OUTPUT] + Name stdout + Match * +``` + {% endtab %} -{% endtabs %} +{% endtabs %} \ No newline at end of file diff --git a/pipeline/inputs/udp.md b/pipeline/inputs/udp.md index e95faf9f2..d93fdaeb5 100644 --- a/pipeline/inputs/udp.md +++ b/pipeline/inputs/udp.md @@ -25,40 +25,25 @@ In order to receive JSON messages over UDP, you can run the plugin from the comm From the command line you can let Fluent Bit listen for _JSON_ messages with the following options: -```bash -$ fluent-bit -i udp -o stdout +```shell +$ ./fluent-bit -i udp -o stdout ``` By default the service will listen an all interfaces (0.0.0.0) through UDP port 5170, optionally you can change this directly, e.g: -```bash -$ fluent-bit -i udp -pport=9090 -o stdout +```shell +$ ./fluent-bit -i udp -port=9090 -o stdout ``` In the example the JSON messages will only arrive through network interface under 192.168.3.2 address and UDP Port 9090. ### Configuration File -In your main configuration file append the following _Input_ & _Output_ sections: +In your main configuration file append the following: {% tabs %} -{% tab title="fluent-bit.conf" %} -```python -[INPUT] - Name udp - Listen 0.0.0.0 - Port 5170 - Chunk_Size 32 - Buffer_Size 64 - Format json - -[OUTPUT] - Name stdout - Match * -``` -{% endtab %} - {% tab title="fluent-bit.yaml" %} + ```yaml pipeline: inputs: @@ -68,10 +53,29 @@ pipeline: chunk_size: 32 buffer_size: 64 format: json + outputs: - name: stdout match: '*' ``` + +{% endtab %} +{% tab title="fluent-bit.conf" %} + +```text +[INPUT] + Name udp + Listen 0.0.0.0 + Port 5170 + Chunk_Size 32 + Buffer_Size 64 + Format json + +[OUTPUT] + Name stdout + Match * +``` + {% endtab %} {% endtabs %} @@ -79,27 +83,38 @@ pipeline: Once Fluent Bit is running, you can send some messages using the _netcat_: -```bash +```shell $ echo '{"key 1": 123456789, "key 2": "abcdefg"}' | nc -u 127.0.0.1 5170 ``` In [Fluent Bit](http://fluentbit.io) we should see the following output: -```bash -$ bin/fluent-bit -i udp -o stdout -f 1 -Fluent Bit v2.x.x -* Copyright (C) 2015-2022 The Fluent Bit Authors +```shell +$ ./fluent-bit -i udp -o stdout -f 1 + +Fluent Bit v4.0.3 +* Copyright (C) 2015-2025 The Fluent Bit Authors * Fluent Bit is a CNCF sub-project under the umbrella of Fluentd * https://fluentbit.io -[2023/07/21 13:01:03] [ info] [fluent bit] version=2.1.7, commit=2474ccc759, pid=9677 -[2023/07/21 13:01:03] [ info] [storage] ver=1.2.0, type=memory, sync=normal, checksum=off, max_chunks_up=128 -[2023/07/21 13:01:03] [ info] [cmetrics] version=0.6.3 -[2023/07/21 13:01:03] [ info] [ctraces ] version=0.3.1 -[2023/07/21 13:01:03] [ info] [input:udp:udp.0] initializing -[2023/07/21 13:01:03] [ info] [input:udp:udp.0] storage_strategy='memory' (memory only) -[2023/07/21 13:01:03] [ info] [output:stdout:stdout.0] worker #0 started -[2023/07/21 13:01:03] [ info] [sp] stream processor started +______ _ _ ______ _ _ ___ _____ +| ___| | | | | ___ (_) | / || _ | +| |_ | |_ _ ___ _ __ | |_ | |_/ /_| |_ __ __/ /| || |/' | +| _| | | | | |/ _ \ '_ \| __| | ___ \ | __| \ \ / / /_| || /| | +| | | | |_| | __/ | | | |_ | |_/ / | |_ \ V /\___ |\ |_/ / +\_| |_|\__,_|\___|_| |_|\__| \____/|_|\__| \_/ |_(_)___/ + + +[2025/07/01 14:44:47] [ info] [fluent bit] version=4.0.3, commit=f5f5f3c17d, pid=1 +[2025/07/01 14:44:47] [ info] [storage] ver=1.5.3, type=memory, sync=normal, checksum=off, max_chunks_up=128 +[2025/07/01 14:44:47] [ info] [simd ] disabled +[2025/07/01 14:44:47] [ info] [cmetrics] version=1.0.3 +[2025/07/01 14:44:47] [ info] [ctraces ] version=0.6.6 +[2025/07/01 14:44:47] [ info] [input:mem:mem.0] initializing +[2025/07/01 14:44:47] [ info] [input:mem:mem.0] storage_strategy='memory' (memory only) +[2025/07/01 14:44:47] [ info] [sp] stream processor started +[2025/07/01 14:44:47] [ info] [engine] Shutdown Grace Period=5, Shutdown Input Grace Period=2 +[2025/07/01 14:44:47] [ info] [output:stdout:stdout.0] worker #0 started [0] udp.0: [[1689912069.078189000, {}], {"key 1"=>123456789, "key 2"=>"abcdefg"}] ``` @@ -107,4 +122,4 @@ Fluent Bit v2.x.x When receiving payloads in JSON format, there are high performance penalties. Parsing JSON is a very expensive task so you could expect your CPU usage increase under high load environments. -To get faster data ingestion, consider to use the option `Format none` to avoid JSON parsing if not needed. +To get faster data ingestion, consider to use the option `Format none` to avoid JSON parsing if not needed. \ No newline at end of file diff --git a/pipeline/inputs/windows-event-log-winevtlog.md b/pipeline/inputs/windows-event-log-winevtlog.md index a2e92f0d8..4a9035239 100644 --- a/pipeline/inputs/windows-event-log-winevtlog.md +++ b/pipeline/inputs/windows-event-log-winevtlog.md @@ -34,8 +34,24 @@ Note that if you do not set _db_, the plugin will tail channels on each startup. Here is a minimum configuration example. {% tabs %} +{% tab title="fluent-bit.yaml" %} + +```yaml +pipeline: + inputs: + - name: winevtlog + channels: Setup,Windows PowerShell + interval_sec: 1 + db: winevtlog.sqllite + outputs: + - name: stdout + match: '*' +``` + +{% endtab %} {% tab title="fluent-bit.conf" %} -```python + +```text [INPUT] Name winevtlog Channels Setup,Windows PowerShell @@ -46,20 +62,7 @@ Here is a minimum configuration example. Name stdout Match * ``` -{% endtab %} -{% tab title="fluent-bit.yaml" %} -```yaml -pipeline: - inputs: - - name: winevtlog - channels: Setup,Windows PowerShell - interval_sec: 1 - db: winevtlog.sqllite - outputs: - - name: stdout - match: '*' -``` {% endtab %} {% endtabs %} @@ -79,9 +82,9 @@ For further details, please refer to [the MSDN doc](https://learn.microsoft.com/ If you want to do a quick test, you can run this plugin from the command line. -```bash -$ fluent-bit -i winevtlog -p 'channels=Setup' -p 'Read_Existing_Events=true' -o stdout +```shell +$ ./fluent-bit -i winevtlog -p 'channels=Setup' -p 'Read_Existing_Events=true' -o stdout ``` Note that `winevtlog` plugin will tail channels on each startup. -If you want to confirm whether this plugin is working or not, you should specify `-p 'Read_Existing_Events=true'` parameter. +If you want to confirm whether this plugin is working or not, you should specify `-p 'Read_Existing_Events=true'` parameter. \ No newline at end of file diff --git a/pipeline/inputs/windows-event-log.md b/pipeline/inputs/windows-event-log.md index 4a6941a40..da7500130 100644 --- a/pipeline/inputs/windows-event-log.md +++ b/pipeline/inputs/windows-event-log.md @@ -21,7 +21,26 @@ Note that if you do not set _db_, the plugin will read channels from the beginni Here is a minimum configuration example. -```python +{% tabs %} +{% tab title="fluent-bit.yaml" %} + +```yaml +pipeline: + inputs: + - name: winlog + channels: setup,Windows Powershell + interval_sec: 1 + db: winlog.sqlite + + outputs: + - name: stdout + match: '*' +``` + +{% endtab %} +{% tab title="fluent-bit.conf" %} + +```text [INPUT] Name winlog Channels Setup,Windows PowerShell @@ -33,12 +52,15 @@ Here is a minimum configuration example. Match * ``` +{% endtab %} +{% endtabs %} + Note that some Windows Event Log channels (like `Security`) requires an admin privilege for reading. In this case, you need to run fluent-bit as an administrator. ### Command Line If you want to do a quick test, you can run this plugin from the command line. -```bash -$ fluent-bit -i winlog -p 'channels=Setup' -o stdout -``` +```shell +$ ./fluent-bit -i winlog -p 'channels=Setup' -o stdout +``` \ No newline at end of file diff --git a/pipeline/inputs/windows-exporter-metrics.md b/pipeline/inputs/windows-exporter-metrics.md index f1b5ed126..a5add77ac 100644 --- a/pipeline/inputs/windows-exporter-metrics.md +++ b/pipeline/inputs/windows-exporter-metrics.md @@ -6,12 +6,15 @@ description: >- # Windows Exporter Metrics -[Prometheus Windows Exporter](https://github.com/prometheus-community/windows_exporter) is a popular way to collect system level metrics from microsoft windows, such as CPU / Disk / Network / Process statistics. Fluent Bit 1.9.0 includes windows exporter metrics plugin that builds off the Prometheus design to collect system level metrics without having to manage two separate processes or agents. +[Prometheus Windows Exporter](https://github.com/prometheus-community/windows_exporter) is a popular way to collect system level metrics from microsoft windows, such as CPU / Disk / Network / Process statistics. Included since Fluent Bit 1.9.0, windows exporter metrics plugin builds off the Prometheus design to collect system level metrics without having to manage two separate processes or agents. -The initial release of Windows Exporter Metrics contains a single collector available from Prometheus Windows Exporter and we plan to expand it over time. +The initial release of Windows Exporter Metrics contains a single collector available from Prometheus Windows Exporter. + +{% hint style="warning" %} **Important note:** Metrics collected with Windows Exporter Metrics flow through a separate pipeline from logs and current filters do not operate on top of metrics. +{% endhint %} ## Configuration @@ -73,7 +76,39 @@ This input always runs in its own [thread](../../administration/multithreading.m In the following configuration file, the input plugin _windows_exporter_metrics collects _metrics every 2 seconds and exposes them through our [Prometheus Exporter](../outputs/prometheus-exporter.md) output plugin on HTTP/TCP port 2021. -``` +{% tabs %} +{% tab title="fluent-bit.yaml" %} + +```yaml +# Node Exporter Metrics + Prometheus Exporter +# ------------------------------------------- +# The following example collect host metrics on Linux and expose +# them through a Prometheus HTTP end-point. +# +# After starting the service try it with: +# +# $ curl http://127.0.0.1:2021/metrics +# +service: + flush: 1 + log_level: info + +pipeline: + inputs: + - name: windows_exporter_metrics + tag: node_metrics + scrape_interval: 2 + + outputs: + - name: prometheus_exporter + match: node_metrics + port: 2021 +``` + +{% endtab %} +{% tab title="fluent-bit.conf" %} + +```text # Node Exporter Metrics + Prometheus Exporter # ------------------------------------------- # The following example collect host metrics on Linux and expose @@ -97,14 +132,15 @@ In the following configuration file, the input plugin _windows_exporter_metrics match node_metrics host 0.0.0.0 port 2021 - - ``` +{% endtab %} +{% endtabs %} + You can test the expose of the metrics by using _curl:_ -```bash -curl http://127.0.0.1:2021/metrics +```shell +$ curl http://127.0.0.1:2021/metrics ``` ### Service where clause @@ -122,12 +158,13 @@ Here is how these parameters should work: `we.service.where` is handled as a raw WHERE clause. For example, when a user specifies the parameter as follows: -``` +```text we.service.where Status!='OK' ``` This creates a WMI query like so: -``` + +```sql SELECT * FROM Win32_Service WHERE Status!='OK' ``` @@ -140,18 +177,19 @@ If multiple key-value pairs are specified, the values will be concatenated with Also, if the values contain `%` character then a `LIKE` operator will be used in the clause instead of the `=` operator. When a user specifies the parameter as follows: -```python +```text we.service.include {"Name":"docker","Name":"%Svc%", "Name":"%Service"} ``` The parameter will be interpreted as: -``` + +```text (Name='docker' OR Name LIKE '%Svc%' OR Name LIKE '%Service') ``` The WMI query will be called with the translated parameter as: -``` +```sql SELECT * FROM Win32_Service WHERE (Name='docker' OR Name LIKE '%Svc%' OR Name LIKE '%Service') ``` @@ -163,18 +201,19 @@ If multiple key-value pairs are specified, the values will be concatenated with Also, if the values contain `%` character then a `LIKE` operator will be used in the translated clause instead of the `!=` operator. When a user specifies the parameter as follows: -```python +```text we.service.exclude {"Name":"UdkUserSvc%","Name":"webthreatdefusersvc%","Name":"XboxNetApiSvc"} ``` The parameter will be interpreted as: -``` + +```sql (NOT Name LIKE 'UdkUserSvc%' AND NOT Name LIKE 'webthreatdefusersvc%' AND Name!='XboxNetApiSvc') ``` The WMI query will be called with the translated parameter as: -``` +```sql SELECT * FROM Win32_Service WHERE (NOT Name LIKE 'UdkUserSvc%' AND NOT Name LIKE 'webthreatdefusersvc%' AND Name!='XboxNetApiSvc') ``` @@ -183,14 +222,14 @@ SELECT * FROM Win32_Service WHERE (NOT Name LIKE 'UdkUserSvc%' AND NOT Name LIKE `we.service.where`, `we.service.include`, and `we.service.exclude` can all be used at the same time subject to the following rules. 1. `we.service.include` translated and applied into the where clause in the service collector -1. `we.service.exclude` translated and applied into the where clause in the service collector +2. `we.service.exclude` translated and applied into the where clause in the service collector 1. If the `we.service.include` is applied, translated `we.service.include` and `we.service.exclude` conditions are concatenated with `AND`. -1. `we.service.where` is just handled as-is into the where clause in the service collector . +3. `we.service.where` is just handled as-is into the where clause in the service collector . 1. If either of the above parameters is applied, the clause will be applied with `AND (` _the value of `we.service.where`_ `)`. For example, when a user specifies the parameter as follows: -``` +```text we.service.include {"Name":"docker","Name":"%Svc%", "Name":"%Service"} we.service.exclude {"Name":"UdkUserSvc%","Name":"XboxNetApiSvc"} we.service.where NOT Name LIKE 'webthreatdefusersvc%' @@ -198,13 +237,12 @@ we.service.where NOT Name LIKE 'webthreatdefusersvc%' The WMI query will be called with the translated parameter as: -``` +```sql SELECT * FROM Win32_Service WHERE (Name='docker' OR Name LIKE '%Svc%' OR Name LIKE '%Service') AND (NOT Name LIKE 'UdkUserSvc%' AND Name!='XboxNetApiSvc') AND (NOT Name LIKE 'webthreatdefusersvc%') ``` - ## Enhancement Requests Our current plugin implements a sub-set of the available collectors in the original Prometheus Windows Exporter, if you would like that we prioritize a specific collector please open a Github issue by using the following template:\ \ -\- [in_windows_exporter_metrics](https://github.com/fluent/fluent-bit/issues/new?assignees=\&labels=\&template=feature_request.md\&title=in_windows_exporter_metrics:%20add%20ABC%20collector) +\- [in_windows_exporter_metrics](https://github.com/fluent/fluent-bit/issues/new?assignees=\&labels=\&template=feature_request.md\&title=in_windows_exporter_metrics:%20add%20ABC%20collector) \ No newline at end of file