Skip to content

multiline: Add a description for the buffer limit of multiline #2095

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 1 commit into from
Oct 10, 2025
+17 −0
Merged

multiline: Add a description for the buffer limit of multiline #2095

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
17 changes: 17 additions & 0 deletions administration/configuring-fluent-bit/multiline-parsing.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -255,6 +255,23 @@ The multiline parser is a very powerful feature, but it has some limitations tha
- The multiline parser isn't affected by the `buffer_max_size` configuration option, allowing the composed log record to grow beyond this size. The `skip_long_lines` option won't be applied to multiline messages.
- It's not possible to get the time key from the body of the multiline message. However, it can be extracted and set as a new key by using a filter.

### Additional configuration for buffer limit of multiline

Fluent Bit now supports a configuration key to **limit the memory used during multiline concatenation**.

| Property | Description | Default |
| ------------------------ | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------- |
| `multiline_buffer_limit` | Sets the maximum size of the in-memory buffer used while assembling a multiline message. When the accumulated size exceeds this limit, the message is truncated and a `multiline_truncated: true` metadata field is attached to the emitted record. A value of `0` disables the limit. Accepts unit suffixes like `KiB`, `MiB`, or `GiB`. | `2MiB` |

```text
service:
multiline_buffer_limit 2MiB # default; limit concatenated multiline message size
```

If the limit is reached, Fluent Bit flushes the partial record with `multiline_truncated: true` metadata immediately to prevent Out-Of-Memory (OOM) conditions.
Copy link
Contributor

@lecaros lecaros Oct 7, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

does this limit work as a soft-limit for the current record being appended? or is the ml record flushed without appending the current record?
what happens to the remaining records intended to be appended to the ml record? are they discarded or flushed as individual records?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

does this limit work as a soft-limit for the current record being appended? or is the ml record flushed without appending the current record?
what happens to the remaining records intended to be appended to the ml record? are they discarded or flushed as individual records?

This works as a hard limit and the remaining records will not be concatenated.
They will be flush as individual record(s).

This option ensures predictable memory usage in Kubernetes or other constrained environments,
particularly when using built-in parsers such as `cri` or `docker`.

## Get structured data from multiline message

Fluent-bit supports the `/pat/m` option. It allows `.` matches a new line, which can be used to parse multiline logs.
Expand Down