Amazon S3: Mismatch when reading HTTP header from GCS

[gouyelliot]

This commit includes the HTTP separator when looking for the header values in the response payload, to avoid mismatching header ending with the same name. See the linked issue for more info and example.

Fixes #8790

---

Testing
Before we can approve your change; please submit the following in a comment:

• Example configuration file for the change
• Debug log output from testing the change

• Attached Valgrind output that shows no leaks or memory corruption was found

If this is a change to packaging of containers or native binaries then please confirm it works for all targets.

• Run local packaging test showing all targets (including any new ones) build.
• Set ok-package-test label to test for all targets (requires maintainer to do).

Documentation

• Documentation required for this feature

Backporting

• Backport to latest stable release.

---

Fluent Bit is licensed under Apache 2.0, by submitting this pull request I understand that this code will be released under the terms of that license.

Summary by CodeRabbit

• Bug Fixes
  • Improved HTTP header parsing so Transfer-Encoding, Content-Length, and Connection headers are reliably detected only at line boundaries, preventing false matches and ensuring correct extraction of header values.

[ghost]

Hey @PettitWesley, can we get this PR reviewed and merged? Currently, there is no GCS Output Plugin but this fix could help as a workaround to upload to GCS bucket meanwhile.

[sunhubs]

why such a bug fix for function is not merged as soon as possible

[ghost]

@PettitWesley @sparrc @singholt @swapneils

Bumping up for review

[vladst3f]

still relevant and needed !

[Daniel-Vaz]

+1 for this

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: aa82a1ab-5ff6-4eb2-9294-4505e4b8cbad

📥 Commits

Reviewing files that changed from the base of the PR and between a1d6fb1 and 7bbc002.

📒 Files selected for processing (1)

• src/flb_http_client.c

🚧 Files skipped from review as they are similar to previous changes (1)

• src/flb_http_client.c

---

📝 Walkthrough

Walkthrough

The HTTP client now matches selected headers only at CRLF-prefixed line boundaries. header_lookup() advances to the header value immediately after a match.

Changes

HTTP Client Header Lookup Fix

Layer / File(s)	Summary
Pointer advancement in header parsing src/flb_http_client.c	header_lookup() advances the search pointer by header_len immediately after a match.
Restrict header pattern matching to CRLF-prefixed lines src/flb_http_client.c	Transfer-Encoding, Content-Length, and Connection searches now include a leading \r\n and use updated header lengths.

Estimated code review effort: 2 (Simple) | ~8 minutes

Suggested reviewers: cosmo0920

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title clearly identifies the Amazon S3 and GCS HTTP-header mismatch addressed by the changes.
Linked Issues check	✅ Passed	The changes prevent suffix matches such as x-goog-stored-content-length from being read as Content-Length, meeting issue #8790.
Out of Scope Changes check	✅ Passed	The changes are limited to header_lookup matching and related HTTP header lookups required to fix issue #8790.
Docstring Coverage	✅ Passed	Docstring coverage is 100.00% which is sufficient. The required threshold is 80.00%.

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[gouyelliot]

I can confirm that this bug is still present in 5.0.3. I rebased my branch on the commit of the release.

[tristanveness]

Can confirm, the bug is still present in fluentbit 5.0.9.

Without this patch, fluentbit writing to a GCS bucket using the S3 output filter doesn't work.

This patch needs to be merged.

[coderabbitai]

Note

GitHub couldn't provide a complete incremental comparison for this pull request, so CodeRabbit is performing a full review instead. This review may take a little longer.